Synthetic Data Engine

2. Synthetic Data Engine synth_hands.py — perfect finger labels

A HaMeR-style two-hand MANO renderer whose projected 2D labels land exactly on the rendered joints (including curled fingers and fingertips). This is the core leverage: it fixes the wild finger-splay that no amount of resolution or MediaPipe distillation could.

• Geometry: smplx.MANOLayer (use_pca=False, flat_hand_mean=False); 21 OpenPose joints = MANO joints re-ordered + 5 fingertip vertices.
• Rendering: pyrender EGL off-screen; labels computed in the OpenCV camera frame (X-right / Y-down / +Z-forward), mesh flipped diag(1,-1,-1) for pyrender.
• Domain randomization: COCO backgrounds, skin tones + gloves, randomized lighting / focal / pose, motion blur.
• 8-frame smooth clips provide temporal + smoothness signal and single-hand diversity (present negatives).
• Visibility comes from the render depth test.

2.1 Three iterations (each A/B-promoted)

Every cell: the red/blue skeleton stays exactly anchored on the true joints even when the hand is foreshortened, motion-blurred, or partly out of frame.

2.2 Real-MANO motion replay synth_replay.py

Problem with v1–v3: randomly sampled poses + crude 2-keyframe trajectories are 3D-implausible (they wobble). Fix: replay real recorded two-hand MANO sequences (ARCTIC / H2O), which are naturally 3D-plausible and temporally coherent, then re-render with high quality + randomizable appearance while keeping labels exact.

The validated conversion chain (test_replay_align.py → reprojection residual 0.000 px):

real MANO (go_world aa, hand_pose OFFSET aa, betas) --[+hand_mean]--> smplx MANOLayer FK (world)
  --> root at zarr world wrist --> transform_world_to_camera (same helper as project_hand)
  --> camera-frame verts --> pyrender IntrinsicsCamera (real K) 2x supersampled render --> perfect uv labels

Gotcha we hit: smplx.MANOLayer takes rotation matrices and cannot add the axis-angle hand_mean; the zarr stores a mean-relative offset. Feeding the offset directly → FK off by 30 mm. Adding hand_mean back → 0.000 mm exact reproduction. That was the key fix.

2.3 SYNTH-v5 — occlusion + appearance + continuous supervision

The augmentation survey's #1 finding: the root cause of "flying fingers" is that synthetic hands were never occluded — the model never learned where a hidden fingertip should go. v5 (render_replay_clips_v5.py) adds three label-safe frontiers on top of replay:

1. 1. Occlusion — procedural occluders (cable / box / sphere / cylinder, ~45% of clips) placed between hand and camera, rendered into the same scene.
2. 2. Appearance — per-hand material randomization (8 glove colors + skin + roughness jitter) and a tapered forearm stub (removes floating hands).
3. 3. Depth visibility (two-pass) — each hand is re-rendered alone for its own depth baseline; only a foreign surface (occluder / other hand) closer than the hand's own surface counts as occlusion → thickness cancels out and wrists / knuckles are never mis-flagged (fixed a single-pass bug that flagged the wrist, vis 0.33 → 0.95).

Crucially, this obeys the tracker semantics from §1.6: occ is visualization-only and never enters the loss; occluded-but-in-frame joints stay under continuous coordinate supervision.

Takeaway — Perfect, label-safe synthetic data — real motion, occlusion, gloves, continuous supervision — is the lever that directly attacks finger-splay and occlusion hallucination without a noisy teacher.

2.4 Training data — representative examples

The Stage-1 mix (§1.8) blends four labeled families into one schema. To make the supervision auditable, below is a curated, quality-filtered look at each source with every label drawn on — sampled straight through the production dataloaders, degenerate/black/mislabeled crops discarded. For the two GT-MANO sources we overlay both the 2D per-finger skeleton and the MANO mesh (blue = anatomical left, salmon = right, the Somantis standard); for the wild + interacting-hands sources we overlay the 2D skeleton. Colors follow the per-finger palette (thumb/index/middle/ring/pinky each a distinct hue).

SYNTH — perfect GT, rendered two-hand MANO (render_replay_clips_v5.py). The rendered pixels are the MANO mesh, so the projected 2D labels land exactly on every joint — including curled, gloved, occluder-hidden, and strongly foreshortened fingers where a real teacher would splay. This is the core leverage of the pipeline: perfect finger labels with heavy domain randomization (COCO backgrounds, glove/skin materials, forearm stubs, motion blur, procedural occluders).

URDS — ARCTIC + H2O lab GT-MANO on real egocentric frames (cache_urds_temporal.py). Real first-person capture with exact GT MANO; the FK-with-hand_mean chain reprojects at 0.000 px, so both the 2D skeleton and the mesh sit precisely on the real hands through bimanual manipulation, tool occlusion (scissors, pen), and pointing/grasping gestures. This is the non-negotiable lab anchor (loc=1). (Left-hand MANO carries the known ~11 mm intrinsic left/right-model asymmetry — the mesh is a hair looser on left hands than on right, which is honest and does not affect the exact 2D labels.)

WILD — COCO-WholeBody, MediaPipe pseudo-labels (dataset_wild.py). In-the-wild appearance and domain coverage (bare, gloved, two-hand, clutter, unusual scenes) that the lab/synthetic sources lack. These are pseudo-labels, not GT: even the clean examples shown here are noisier than the GT sources, and MediaPipe splays on the hardest hands (the known ceiling from §1.8) — which is exactly why WILD supplies wild localization + appearance signal and negatives rather than finger-precision. The HaMeR 2D-teacher re-distillation of this source is complete and shipped: the current champion (v11) trains on the HaMeR-relabeled wild set (§2.5).

InterHand2.6M — lab GT 2D (dataset_interhand.py). Exact GT keypoints with the field's richest two-hand interaction and single-hand pose ROM (thumbs-up, pinch, open splay, interlocked hands). This is the localization anchor for touching/crossing hands that neither ARCTIC/H2O nor the synthetic engine covers as densely.

Takeaway — Two GT-MANO sources (synthetic + lab, labels drawn on and verified to land on the joints), one dense two-hand-interaction GT-2D source, and one wide-but-noisier wild pseudo-label source — mixed by mass. The supervision that the model actually sees is auditable, and where a source is weaker (wild pseudo-labels) we show it and say so.

2.5 WILD teacher re-distillation — HaMeR-2D replaces MediaPipe (shipped as champion v11)

The wild source above had two systematic teacher flaws that no amount of training could fix downstream:

1. 1. Finger splay on grasps — on curled/grasping hands MediaPipe emits a flat, splayed fan where the fingers are actually wrapped around an object. Distilling that teaches the model to splay.
2. 2. Handedness by selfie-flip heuristic — MediaPipe's left/right came from a blanket mirror assumption, which is systematically wrong on mirrored/edited web images. Re-deriving handedness from the COCO-WholeBody annotation hand boxes (with a run-HaMeR-both-ways chirality fallback where no box matches) found ~5% of hands carried the wrong handedness label.

The fix (hand2d/hamer_relabel_wild.py): re-label the same wild image set with HaMeR-2D — official HaMeR MANO joints projected to the full image with HaMeR's own camera, exactly like the _bench referee — with handedness from the COCO annotation side. MediaPipe is kept as a cross-check only: hands where the two estimators disagree beyond a bbox-normalized tolerance are dropped as ambiguous (~10% of positive images dropped — not worth training on with either label). Output is a drop-in replacement jsonl for the existing dataloader.

The QC sheet is the whole argument in one image: on grasping hands MediaPipe draws flat fans; HaMeR curls the fingers where they actually are.

Status — closed and shipped. The retrain on these labels is v11_wildhamer_36k, now the champion: warm-started from v7, with the wild-teacher swap as the only change. It beat v7 on every GT-frontier axis (overall PCK@0.10 0.829 vs 0.817, TIP-PCK@0.05 0.348 vs 0.328, wins on all five frontier subsets), on all lab sets (mean5 0.659 vs 0.643), improved calibration (ECE 0.052 vs 0.060, AUROC 0.904 vs 0.893), kept presence specificity (0.994), and passed the wild-stability gate with improvement (jitter p95 −10%, teleports −11%) — the first champion promoted through the full two-leg gauntlet (§4.3). The one number that dropped, wild-vs-MP (0.459 vs 0.562), is the expected signature of no longer imitating the MediaPipe teacher — and this time every gating instrument moved the other way (the v10 red-flag pattern is explicitly absent). Visual audit: local_2d/audit/v11_promotion/.