Stage-1 · Two-Hand 2D Tracker

1. Stage-1 · Two-Hand 2D Tracker HandBidirViT

Stage-1 is a single-model, two-hand, fixed-2-slot, bidirectional-temporal 2D keypoint regressor. Data flow: clip → frozen ViT backbone (per frame) → root heatmap → bidirectional temporal transformer → decoupled representation heads → absolute 2D reconstruction.

1.1 Backbone — HaMeR ViTPose-H (frozen, optionally unfreeze last blocks)

Why ViTPose-H: an earlier ImageNet-ResNet50 hub capped wild localization at PCK-vs-MediaPipe ≈ 0.086 — its features simply don't transfer to hands in the wild. ViTPose-H is pretrained on large-scale hand data and warm-starts cleanly (389/389), so the entire lineage reuses the same vendored, timm-free ViT.

1.2 Root heatmap head

Token map is bilinearly upsampled to 48×48 → a fresh conv head emits a per-hand root heatmap → matched-grid soft-argmax on a [-pad, 1+pad] grid (hm_pad=0.1, so the wrist may sit slightly out of frame) → square-canvas-normalized absolute wrist root_abs.

1.3 Bidirectional temporal transformer

Bidirectional attention lets each frame see the full window, so motion-blur and occlusion frames are filled from both temporal directions — far steadier than an autoregressive pass.

1.4 Decoupled representation heads

The model does not regress absolute pixel coordinates. It regresses a set of decoupled, differentiably-decoded quantities, so a localization error never contaminates the gesture:

Reconstruction (also the lift interface): uv[0] = root_uv; uv[k] = root_uv + s · j_rel[k-1]. Joints follow the 21-point OpenPose/HaMeR order, identical to the repo's MANO FK convention — no re-indexing needed downstream.

Local-finger residual head (v10 — promoted on lab GT, rolled back on wild evidence). The temporal token that feeds jrel_head is a mean-pooled global image feature — a resolution bottleneck for fingertip precision (a fingertip is a few pixels; a pooled 256-token summary blurs it). v10 adds a +3.9M-param local-feature residual head: it grid_samples the ViT feature map at the predicted root and emits a residual correction to j_rel from genuinely local features. It won every lab-GT gate (§4.3: ARCTIC-VAL PCK@0.10 +0.008, GT-frontier tips +0.016, foreshortened +0.021, truncated PCK@0.10 +0.043) and was promoted — and the user immediately saw wild regressions on real footage (2D jitter, degraded tracking). A rigorous A/B confirmed the regression lives in the tail of the temporal distribution, invisible to per-frame median-based lab metrics: whole-hand teleport spikes +60% under an identical inference gate, phantom corner hands on 16.7% of guitar frames, teleports at raw confidence 0.96+. The watch items that had been dismissed at promotion time — ECE 0.061→0.097, specificity 0.994→0.982, wild-vs-MP −1.1pt — were real signal. The champion was rolled back to v7 within hours, and v10 stays in the lineage honestly labeled lab-better / wild-worse. The idea itself remains viable — local features for fingertip precision attack a real bottleneck — pending a variant that also passes the new wild-stability gate (§4.3). Full audit trail: local_2d/audit/v10_wild_regression/ (side-by-side videos, stills, spike traces). Two successors have since run the gauntlet: v12 (local head + the v11 HaMeR wild labels) again won every GT-crop axis and was again rejected — this time before promotion — by the wild-stability flicker gate and a freeze audit that caught a guard-manufactured frozen phantom (§4.3); and v13, the presence-isolated control (local path learns from detached trunk/backbone features, so presence receives exactly the champion's gradients), which was also not promoted but settled the science (§4.3 part 5): the GT tip gains survived full gradient isolation (tips +0.9pt, near-camera tips +1.4/+2.7pt — though the seed controls later placed these inside the ±1.9pt run-variance band, §7), wild stability passed and the freeze audit came back clean — yet GT-frontier presence failed harder than v12 (phantom second hands on single-hand frames, monotone worse across v11 → v12 → v13). Since v13's local path is fully gradient-isolated, that result refutes the v10/v12 "gradient leakage" diagnosis; the remaining suspects are continued-training drift and loss-level coupling, and the decisive control (v14: identical continued training, no local head) is running now. The local-head line is blocked, not dead — the tip win is proven real; what is missing is a presence-safe training recipe.

1.5 Fixed 2-slot output — presence-gated, NOT detection

The output is always 2 slots: slot0 = anatomical left hand, slot1 = right hand, each gated by present. This is deliberately not detection — slots are bound to anatomical identity, which sidesteps two-hand matching and ID jitter entirely.

1.6 Persistent-latch tracking (tracker, not detector)

The core tracking semantics: output is continuous. Once a hand is being tracked, coordinates stay continuous through occlusion; the model flags only genuine off-frame.

• The occlusion flag occ is visualization-only (the point is hidden in the overlay) and never enters the loss.
• The trained vis field is in-frame only → a joint that is occluded but still inside the frame is supervised with continuous coordinates (synthetic data has perfect GT, teaching the model "the hand doesn't teleport — keep emitting the hidden joint").
• present stays 1 while the hand is in the sequence; when it leaves the frame, the in-frame flag drops to 0 and the model learns to flag off-camera.

Presence-recall upgrade (inference-side latch, infer.persistent_track). The plain present-gate treated present as a per-frame confidence, so hard frames (occlusion, blur, frame-edge) dipped below threshold and the whole hand vanished — a tracker must not do that. Three original refinements (1–3), each verified against the OOD true-negative clips so specificity is not traded away, plus two later stability layers (4–5) added by the v10/v12 audits:

1. 1. Hysteresis — a track enters at confidence 0.5 (2 consecutive frames) but is held down to 0.25, so a confident track whose confidence merely dips at its head/tail no longer loses those frames (a genuinely absent hand sits near 0.01, far below the hold band).
2. 2. Sustained mid-confidence latch — some real hands never cross 0.5 (a strumming hand half-hidden by a mic hovers at 0.35–0.50 for the whole clip). A sliding ~1 s window latches the hand where the window's median confidence and its fraction-above-hold both qualify; true-negative phantoms (median ~0.1–0.15 with brief blips) fail both conditions. No fabrication — this only un-gates the model's own continuous coordinates.
3. 3. Joint-based off-frame test — the old wrist-only test dropped hands whose wrist sat barely past the letterbox bound while the fingers were plainly visible. A hand is now off-frame only when fewer than 6 of its 21 joints are inside the frame bounds.
4. 4. Anti-teleport guard (added in the v10 rollback hardening, §4.3) — the latch's recall wins came with a cost the v10 audit exposed: a sub-threshold frame it un-gates can carry a wrist that jumps. Within a latched track, a sub-threshold frame whose wrist moves more than 8% of the square canvas (teleport_gate=0.08) from the last kept position is now held at the last kept coordinates — continuous, no fabrication, no teleport (confident frames are always accepted). Net effect on the 6-clip wild stability set: teleports 94 → 76/min vs the pre-latch config, with the latch's recall wins (guitar-R 100%) fully kept. Guard hardening (M4/M5, from the v12 freeze audit, §4.3): the guard now runs before the off-frame test, so a single-frame glitch can no longer masquerade as an off-camera exit — wild flicker halved (0.201 → 0.110/s); anchor frames must carry confidence ≥ 0.5 (a low-confidence frame can never become the position other frames are held to); holds are capped at 0.4 s, so the guard can never freeze a hand for multi-second stretches; and guard engagement is now telemetered per clip — so a model that games the teleport metric by making the guard freeze wilder raw jumps (exactly what v12 did) shows up as a number, not a hunch.
5. 5. Light root-trajectory optimization (default-ON in the benchmark path) — Dyn-HaMR's one remaining structural advantage over a feed-forward tracker was its global temporal optimization over the whole sequence (§5). We absorbed that idea at ~1/1000 of the cost: a per-track, confidence-weighted Huber data term + velocity/acceleration smoothness, solved over the entire clip, root-only — fingers keep their per-frame best predictions untouched — in 0.2 s per clip. Validated end-to-end on the stability set: teleports −21.5% (68.3 → 53.6/min), jitter p95 −19.8% (23.5 → 18.8 px), flicker and present-rate untouched, GT frontier PCK unchanged (+0.003), and zero fast-motion lag — drum-stroke amplitude 97.7% preserved (local_2d/audit/traj_opt2d/).

Result: both-hand recall on the previously-lost wild clips went guitar-R 0% → 100%, medical 100%, ego_wire 98%, while OOD specificity held — genuinely-absent hands (conductor / speaker true negatives) stay flagged, with no fabricated skeletons. The shipping-config champion stability baseline is now jitter p95 18.83 px / flicker 0.110/s / teleports 53.6/min / present 0.983 (guard M4/M5 + traj-opt; local_2d/audit/wild_stability/), improved from 24.4 / 0.20 / 94 at the start of the campaign — a 23% / 45% / 43% wild-stability improvement arc.

1.7 Loss (head-isolated)

The key design is head isolation: localization terms (root / rootabs / heat / scale / jrel / vis) backprop only on clean loc=1 samples, while the presence/confidence head trains with balanced BCE across all samples including negatives — that is where calibration comes from. Root uses a three-tier weight (in-frame 1.0, out-of-frame 0.3, absent 0.1); jrel drops extreme |target|>50 values (a near-camera hand has tiny s, so (uv-root)/s explodes). Weights: root 5.0, rootabs 2.0, heat 1.0, scale 1.0, jrel 1.0, vis 0.5, present 1.0, smooth 0.05. Total params 639M, trainable 165M (frozen backbone) to 244M (unfreeze-12).

1.8 Dataset integration (Principle 3)

A single MixedWildDataset mixes five source families into one schema by mass:

Key insight: MediaPipe pseudo-labels already splay on hard hands (gloves, blur, foreshortening), so distilling them directly inherits their ceiling. Resolution is not the fix (full 1920×1080 still splays). The real fix is rendering synthetic data with exact 2D labels — see §2.

1.9 Training

OneCycle warm-start dip: warm-start + LR spike makes lab PCK dip around step 2,500, then recover and overtake by 10–12k. Do not conclude before 10k.

1.10 Results — generation-by-generation

lab mean5 = mean PCK across the 5 URDS lab datasets. wild-vs-MP = agreement with MediaPipe (a noisy proxy — it does not measure finger-splay, since MediaPipe itself splays).

† v11's wild-vs-MP is not comparable to earlier rows: v11 no longer trains on the MediaPipe teacher, so teacher-agreement structurally penalizes it — exactly the referee-demotion logic of §4.3 playing out. Every instrument that actually gates (GT frontier, lab, calibration, wild stability) improved, and the v10 red-flag pattern (calibration drift + specificity drop) is explicitly absent.

Total gain: mean5 +0.138 (0.521 → 0.659); wild-vs-MP rose +0.185 (0.377 → 0.562) through v7, then was retired as a comparable axis when the teacher was replaced (†). wild-vs-MP is a drift indicator only — MediaPipe is (partially) our own teacher, so promotion gates run on the GT instruments in §4.3 instead — but the v10 episode sharpened what "indicator" means: v10 beat v7 on every lab-GT axis (overall PCK@0.10 0.817 → 0.822, TIP-PCK@0.05 0.328 → 0.344) while its wild-vs-MP dropped 1.1pt, and that drop turned out to be real tail instability, not teacher bias (§4.3). So teacher-agreement cannot be the gate — but a wild drop that co-occurs with calibration drift must be investigated, never dismissed. Tips remain the honest frontier — see §4.3.

1.11 Occlusion scale-up A/B (v5 → v6 → uf12)

The decisive line of experiments against "hands vanishing / fingers flying under occlusion." A 5-way A/B (raw model + inference-side latching; present measured on 6 wild ego clips):

\*t16's lab is under-reported: its URDS eval ran at T=8 while the model is T=16.

Result: data + training scale-up work across the board, driving occlusion continuity (present) from 71% → 85%. The winner uf12 wins lab mean5 (0.6218, 7/7 domains), ties v6 on wild, and leads on occlusion continuity (85%) → promoted at the time (since superseded by v7, §1.10, which adds the finger-splay synth on the same recipe). T=16 is a valuable negative result: raw present rises slightly (82%) but neither lab nor wild beats uf12 → at this scale, capacity (unfreeze) > context (sequence length).

1.12 Clean-case localization & wild specificity

When fingers are visible (clean lab), green predictions sit almost exactly on the blue/red GT — 2D is near-perfect.

On wild frames: teddy bears / dogs / no-hand images produce no false skeletons (good presence specificity); on real hands the prediction hugs the fingers.

1.13 Honest frontier (ego_wire, first-person)

Left = newer model, right = older. Blue = left hand (gloved), red = right hand (bare, foreshortened reach).

On the single hardest ego_wire clip the two generations are visually close. The clearest gain is a more decisive present confidence on the foreshortened, half-out-of-frame right hand. The remaining hardest frontier is hallucination when fingers are fully occluded — the target of the occlusion work in §2 and the next-step re-distillation in §7.

Takeaway — A single presence-gated 2-slot model, trained head-isolated on mixed data, tracks two hands continuously through occlusion (present 85%) and generalizes to the wild without false detections.