3D Lift · HaMeR + Temporal MANO + 2D-Constrained Solve
3. 3D Lift — real HaMeR on the crop, temporal MANO, 2D-constrained solve
The lift turns 2D keypoints into a consistent 3D MANO hand without moving the 2D overlay (Principle 2). It has three cooperating parts: a real HaMeR MANO regressor on the Stage-1 crop, a distilled temporal MANO student for speed, and an optional 2D-constrained optimization for the tightest overlay fidelity.
3.1 Real HaMeR on the Stage-1 crop (+ recent fixes)
Running real HaMeR on the crop the 2D tracker localizes fixes the classic monocular front/back flip (palm-toward vs. away). Recent stability fixes that made it production-usable:
- Clip-constant median betas → stable hand size (no frame-to-frame shape breathing).
- Savitzky-Golay rotation smoothing → no orientation jitter.
- Depth-divergence guard → rejects implausible per-frame depth jumps.
- Auto per-clip focal → sane intrinsics when the true camera is unknown.
- Velocity-gated per-DOF articulation smoothing → uniform temporal smoothing rounds off fast real finger motion (guitar fretting) exactly like it rounds off jitter. Jitter lives in global-orient + translation; fast articulation lives in hand-pose — so the hand-pose smoothness weight is now gated down per frame-step where the root-relative, palm-normalized Stage-1 finger speed is demonstrably high and Stage-1 visibility is confident (never below a floor, so there is no free jitter). Global-orient and translation stay fully smoothed.
- Fingertip data-term upweight → where Stage-1 visibility on a fingertip is high, that tip's 2D anchor is reliable, so its reprojection weight is boosted (2×) — tips stick to their anchors instead of being averaged away by the smoothness/prior terms.
- Light root-trajectory optimization (§1.6, default-ON in the benchmark path) → upstream of the lift, the Stage-1 root track is now solved once over the whole clip (confidence-weighted Huber + velocity/accel smoothness, root-only, 0.2 s/clip). This absorbs Dyn-HaMR's one remaining structural advantage — global temporal optimization — at ~1/1000 the cost, with teleports −21.5% and jitter p95 −19.8% while fingers keep their per-frame best predictions (drum-stroke amplitude 97.7% preserved).
Measured effect of the velocity gating + tip upweight: fast-motion fingertip lag drops 24–32% (guitar 7.3 → 5.0 px) with measured articulation jitter unchanged — still 3.8–75× lower than Dyn-HaMR across the comparison suites.
3.2 Distilled temporal MANO student (model_lift_temporal.py)
A feed-forward bidirectional-temporal network distilled from HaMeR, taking Stage-1 crops → MANO parameters (global orient 6D in camera frame, 15×6D finger pose, cam_t, clip-pooled betas). It is the fast option — one forward pass instead of a per-crop HaMeR call — and provides the gesture + in-plane orientation prior for the optimizer.
Honest Stage-2 numbers (lab val, n=601): bidir MPJPE 34.6 mm / PA-MPJPE 12.7 mm / orientation 27.5° vs perframe 46.2 / 15.4 / 43.3°, with 6× lower jitter. A single-clip overfit reaches MPJPE 2.5 mm — so the FK and capacity are fine; the student is currently underfit and is the main 3D headroom (scalable with the §2.2 perfect-label replay data).
Key trade-off: the RGB-regressed depth cam_t is discarded downstream — monocular depth is the one unobservable axis. We keep only the reliable gesture + in-plane orientation.
3.3 2D-constrained MANO optimization lift_v2.py
The actual keypoint→MANO conversion is analysis-by-synthesis, not a closed-form inverse: initialize from the student, then optimize full-MANO so its projection sticks to the Stage-1 2D, with depth set by a pointmap.
Stage-1 2D (in-plane, accurate) --+
Temporal MANO prior (gesture init) --+--> Adam ~700 steps on full-MANO (go_world, pose, transl_world) --> consistent 3D MANO
Pointmap (depth / scale) --+| Term | Weight | Role |
|---|---|---|
| reproj → Stage-1 2D (vis-masked, in-plane) | 12.0 | primary — MANO joints project onto the 2D anchors, fingers align |
| pointmap depth anchor (conf-weighted wrist depth) | 3.0 | supplies the unobservable depth / scale axis |
| temporal smoothness (transl vel/acc + pose vel) | 40/80/6 | jitter suppression |
| stay-near-prior (pose close to the student) | 2.0 | regularizes the depth-ambiguous DOF, stops fingers flipping out of plane |
2D overlay is never broken: the left panel = Stage-1 2D anchors (untouched), the right panel = the solved 3D reprojected — the two agree. Validation: 3D-reprojected-to-2D residual 2.6 px (L) / 5.7 px (R) on ego_wire, 2.1 / 5.5 px on ego_desk — all without GT.
3.4 Honest remaining gap
The optimization is a validated PoC, not a large-scale-benchmarked product: it proves the "2D anchor + pointmap depth + MANO prior → consistent 3D" path works (reproj 2.6–5.7 px), but 3D MPJPE is not yet quantified across datasets. The temporal student is underfit at 34.6 mm. Both are addressed by scaling the perfect-label replay data.
HaMeR ego-foreshortening fine-tune (C4) — rejected, an honest negative with a useful reframing. The head-only fine-tune (hand2d/finetune_hamer_fs.py) attacked the lift's weakest crops — strong ego foreshortening — and on GT crops it genuinely worked: foreshortened-subset root-aligned MPJPE −6.2 mm (26.2 → 20.0), global-orientation error halved (24.1° → 12.1°), with the normal subset improving too (local_2d/hamer_ft/evals/). But it regressed the very wild ego clip that motivated it — ego_wire worst-window reprojection 20.5 → 29 px — so it does not ship. Two diagnoses, both actionable: (a) crop-domain gap — the fine-tune trained on GT-joint boxes but deploys on our palm-anchored Stage-1 boxes; any retry must train on deployment-style crops. **(b) The ego_wire worst-window is substantially a Stage-1 2D failure — the 2D scatters over cable clutter there, and no HaMeR checkpoint can fix a bad 2D reference — which reframes that failure class as a Stage-1 data frontier** (§7), not a lift problem. The fine-tuned checkpoint stays available behind an opt-in hamer_ckpt path.
Takeaway — Real HaMeR on the crop fixes the front/back flip; the recent stability fixes remove jitter/breathing; the 2D-constrained solve keeps the overlay pixel-tight while making 3D consistent. The gap is 3D metric accuracy, and we say so.