EgoHand4D · Two-Hand Egocentric Hand-Pose Pipeline

Branch haoran_dev/sota-v1 · Current best model v11_wildhamer_36k (v7 recipe + HaMeR wild teacher, live on API) — the first champion promoted through the full two-leg gauntlet (lab GT + wild stability). The gauntlet's lineage: v10 won every lab-GT gate, was promoted, regressed in the wild tail, and was rolled back within hours — forging the mandatory wild-stability gate (§4.3); v12 then won every GT-crop axis and was rejected before promotion by that same gate plus a new freeze audit (§4.3). Wild-stability arc this campaign: jitter p95 24.4 → 18.8 px, flicker 0.20 → 0.11/s, teleports 94 → 54/min. · Report updated 2026-07-03

0. Overview & Motivation

EgoHand4D is a "2D-first → 4D" two-hand, egocentric hand-pose system. Instead of regressing 3D directly from a single RGB frame (where monocular depth is fundamentally unobservable and errors leak into every joint), we anchor everything on a clean, generalizable 2D tracker, then lift to 3D under strict constraints that keep the 2D overlay intact.

Why egocentric two-hand is hard. Hands are small, fast, self-occluding, frequently gloved, and constantly leaving and re-entering the frame. Both hands touch and cross. Strong perspective foreshortening (a hand reaching toward the camera) breaks appearance priors. Off-the-shelf detectors flicker on identity and hallucinate fingers under occlusion. A pipeline that is honest about these failure modes — rather than papering over them — is the whole point.

Three guiding principles shape every design decision:

1. 1. A pure, generalizable Stage-1 2D tracker trained on _perfect_ data. Stage-1 is 2D-only. It is a tracker, not a detector: it outputs continuous coordinates through occlusion and flags only genuine off-frame. Training labels must be perfect, and the model must generalize far beyond the lab.
2. 2. A 3D lift that must NOT degrade the 2D overlay. Lifting to 3D has to stay 3D-consistent and produce correct gesture/orientation, while the 2D projection continues to sit exactly on the 2D anchors. If the lift moves the keypoints, the lift is wrong.
3. 3. Deeply integrated datasets + a standardized benchmark. Five lab datasets, interacting-hands data, in-the-wild pseudo-labels, and perfectly-labeled synthetic data are mixed into one schema, and every model is judged through one canonical 4-panel benchmark.

What this report covers. Stage-1 2D tracking (§1), the synthetic data engine that makes the labels perfect (§2), the 3D lift (§3), the standardized benchmark, model gallery, and the ground-truth + wild-stability evaluation instruments that now gate every promotion — including the v10 promote-and-rollback episode that forged the wild-stability gate (§4), an honest head-to-head against Dyn-HaMR — referee-based and GT-based (§5), in-the-wild inference (§6), and the evidence-driven campaign status (§7).

Takeaway — Anchor on 2D, lift under constraint, and gate on ground truth and wild stability. Every claim below is reported honestly, including where the pipeline still loses — GT-certified numbers where GT exists, clearly-labeled proxies where it does not, and one promotion (v10) that the process itself caught and reversed within hours.

Progress at a glance

• Stage-1 (2D tracking) — single model · two hands · fixed 2 slots · bidirectional temporal · presence-gated tracker. Best v14d_seedctl_36k, lab mean5 0.6626, GT TIP-PCK@0.05 0.328.
• Synthetic data — a MANO renderer with perfect 2D finger labels, real-motion replay (reproj 0.000 px), and occlusion + continuous supervision to kill finger-splay.
• 3D lift — real HaMeR on the crop (front/back fix) + a distilled temporal MANO student + a 2D-constrained solve keeping the overlay at 2–6 px.
• Benchmark + GT instrument — one canonical 4-panel entry point, a model gallery across 17 clips, a frozen ground-truth gate (GT-2D tip-PCK frontier + GT-3D on 27 windows): RA-MPJPE 18.4 mm, hand length +0.9 cm, swap 0.000 — plus a mandatory wild-stability gate added after the eval process caught and rolled back its own v10 promotion within hours.
• vs Dyn-HaMR — on exact GT: RA-MPJPE 14.9 vs 55.7 mm, PA (Sim(3)) 7.9 vs 27.8 mm, hand length +0.5 vs −7.3 cm, ~100× faster; upstream Dyn-HaMR's optimization converged on 0/14 egocentric clips.

Explore