Standardized Benchmark, Gallery & GT Instrument

4. Standardized Benchmark & Model Gallery

Every model is judged through one canonical entry point: a synchronized 4-panel strip — raw | Stage-1 2D | MANO-AR | 3D + ego-camera orbit — produced by the same renderer the API and CLI emit (benchmark.py). Because the gallery, the API, and the CLI all share this renderer, the gallery is every other inference output.

4.1 The 4-panel entry point

Panel ② uses a per-finger colored skeleton (thumb/index/middle/ring/pinky each a distinct hue, palm bones in the hand's Somantis color so left/right stays legible) with thick anti-aliased bones and outlined joint dots that pop on any background — a single reusable helper (viz_common.draw_hand_skeleton) shared across the pipeline.

4.2 Model iteration gallery

The benchmark gallery is now one grid, one video per clip, every method inside: each of the 23 benchmark clips (8 curated wild/OOD + 9 LineWise industrial-ego + 6 exact-GT lab windows) shows a single task-aligned composite — row 1 (2D tracking) = the current champion's Stage-1 2D skeleton vs MediaPipe 2D in the identical per-finger style; row 2 (3D) = the champion's MANO-AR overlay + 3D orbit + ego-cam; row 3 (3D) = Dyn-HaMR (corrected-K hardened fork, same shared renderer) — each method compared on the task it does. The composites are produced by one driver (hand2d/bench_compare.py) that is champion-aware: when a new model is promoted, --render regenerates every composite with the new champion in row 1, so the gallery tracks the iteration by construction. The lineage table (every promotion/rollback, v10 honestly labeled) stays below the grid; the default champion is read from best_model.json at build time, so the gallery can never default to a demoted model. Clips whose baselines are still computing fall back to the champion 4-panel with an explicit label — the deployed static site never shows a black/broken box.

4.3 GT-based evaluation instrument — the promotion gate

Why we rebuilt evaluation. The wild metric used through §1 (wild-vs-MP) scores agreement with MediaPipe — but MediaPipe pseudo-labels are part of our own training mix (§1.8), so that referee was (partially) our own teacher, not an independent judge: it cannot see improvements past the teacher's ceiling, and it structurally flatters anything that imitates the teacher, splay included. It is now demoted to a drift indicator. Every promotion gate runs on ground truth, through three frozen instruments:

MediaPipe never touches either GT instrument, so teacher bias cannot leak into the gate.

GT-2D baselines — v7 → v11 (champion) (overall on the unbiased random portion; subsets pool the whole slice):

The honest reading: overall 2D localization is strong (PCK@0.10 0.829), and the HaMeR wild relabel (v11) moved every slice — but fingertips remain the frontier: TIP-PCK@0.05 is 0.348 overall and drops further under foreshortening / near-camera / truncation. That is exactly the axis the finger-splay synth (§2), the wild relabel (§2.5), and the local-finger head (§1.4) attack, and this table is where their success or failure shows. (v7 detail rows: results_v7.json; v11: results_v11_36k.json.)

The instrument in action, part 1 — the v10 promotion. The local-finger-head experiment was the first promotion decided end-to-end by this instrument, and it shows why the referee change matters. At its 18k mid-gate v10 trailed v7 (PCK@0.10 0.791 vs 0.817, TIP-PCK@0.05 0.281 vs 0.328); at the full 36k gate it beat v7 on every GT axis — overall PCK@0.10 0.822, TIP-PCK@0.05 0.344 (+0.016), foreshortened tips +0.021, truncated PCK@0.10 +0.043 — and was promoted. Meanwhile its wild-vs-MP dropped 0.011, which was written off at the time as teacher bias (the old teacher-agreement referee scoring a genuinely better model as a regression). (local_2d/frontier_eval/results_v10lf_36k.json; the GT-3D and head-to-head numbers below and in §5.3 are the champion v7's.)

The instrument in action, part 2 — the v10 rollback (the gate gets hardened). The story did not end at promotion, and reporting the second half honestly matters more than the first: within hours of v10 going live, the user's visual check on wild footage caught what every lab number had missed — 2D jitter and degraded tracking. A rigorous A/B on the 6-clip wild set (local_2d/audit/v10_wild_regression/, side-by-side videos, stills, per-frame spike traces) confirmed the regression and localized it in the tail of the temporal distribution, exactly where per-frame, median-based lab metrics are blind:

• Whole-hand teleport spikes +60% from the model alone (112 vs 70 spikes under the identical inference gate; 127 vs 94 per-min in the as-shipped configs) — including spikes at raw confidence 0.96–0.999, which no confidence gate can filter out;
• phantom corner hands on 16.7% of guitar frames (image below);
• raw fingertip acceleration p95 worse on 4 of 6 clips.

Notably, v10's GT-3D metrics were at parity or slightly better than v7's (MPJPE 75.7 vs 75.9 mm, PA-Sim(3) 8.8 vs 9.0 mm — results_ours_v10.json): the regression lived purely in 2D tail stability. Different axes fail independently — no amount of additional lab-3D scoring would have caught it either, which is exactly why the wild-stability gate had to be a separate, mandatory leg of the protocol.

The indicators dismissed at promotion time — ECE 0.061 → 0.097, presence specificity 0.994 → 0.982, wild-vs-MP −1.1pt — had all been pointing at this failure mode. The lesson, now written into the protocol: per-frame lab metrics are blind to tail instability (spikes live in the p95+, not the median), and a wild-referee drop that co-occurs with calibration drift is a red flag for it — never to be dismissed as teacher bias.

The fix, in three parts. (1) The champion was rolled back to v7 (registry + API, same day). (2) An anti-teleport guard was added to infer.persistent_track (§1.6): net wild teleports 94 → 76/min vs the pre-v10 config, with the presence-recall wins (guitar-R 100%) fully kept — the tracker after the episode is more stable than before it. (3) Wild temporal stability is now a mandatory promotion gate: eval_wild_stability.py runs the full user-facing inference pipeline on the fixed 6-clip wild set, and promote_best.py refuses any promotion that does not carry its output. The gated metrics and thresholds:

The complete arc — promoted on lab GT → the user's visual judgment caught the wild tail within hours → the A/B protocol localized it → rollback + tracker guard + a hardened gate — is the instrument working as designed. The system caught and corrected its own bad promotion because the visual check and the A/B protocol did their jobs; the failure mode is now a number, so it cannot recur silently.

The instrument in action, part 3 — the v11 promotion (the two-leg gate passing a good model). The HaMeR wild-teacher retrain (§2.5) was the first candidate to face the full post-v10 gauntlet, and it passed both legs cleanly: every GT-frontier axis up (table above — all five subsets), all lab sets up (mean5 0.659 vs 0.643), calibration improved (ECE 0.052 vs 0.060), presence specificity unchanged (0.994, swap 0.0014), the wild-stability gate passed with improvement (jitter p95 −10%, teleports −11%, flicker/present equal), and a frame-level visual audit clean (local_2d/audit/v11_promotion/ — guitar + ego_wire 4-panels vs v7 references: no phantoms, no teleports, no splay). Its one declining number — wild-vs-MP 0.459 vs 0.562 — is the referee demotion working as intended: the model no longer imitates MediaPipe, and every instrument that actually gates moved the other way. v11 is the shipping champion.

The instrument in action, part 4 — the v12 verdict (an honest negative; the gauntlet working). v12 (ckpt_v12_combo_36k) combined the local-finger head (§1.4) with v11's HaMeR wild labels — and on GT crop accuracy it won everywhere: frontier tips +1.2pt overall (TIP-PCK@0.05 0.359 vs 0.348, PCK@0.10 0.835 vs 0.829), lab +0.2pt, and wild 2D under both referees (HaMeR-ref 0.641 vs 0.596; MP-ref 0.511 vs 0.495 — beating the champion even on the referee it does not train from). It was still rejected, twice over:

1. 1. Flicker hard-gate: +22% — an outright wild-stability gate failure; no judgment call needed.
2. 2. The freeze audit caught a frozen phantom on a moving hand. v12's headline stability number — teleports −27% (49.6 vs 68.3/min) — looked like a win, but the code review had predicted exactly this failure mode (H1): a model with wilder raw jumps whose teleport count improves because the anti-teleport guard converts those jumps into freezes — gate-gaming, not stability. The new guard telemetry (§1.6) confirmed it: 2× guard engagement vs v11 (26 vs 13 held frames on ego_wire), including a 0.67 s hold (20 frames) during ~300 px/frame real hand motion — a skeleton pinned frozen mid-air while the hand moved (local_2d/audit/v12_promotion/freeze_audit.json, pair stills alongside). The teleport "win" was manufactured by the guard.

Diagnosis (as written at the time). Two local-finger variants had now failed with the same signature across two different wild-label sets (v10, v12): when the local head's gradients flow into the shared trunk, presence specificity degrades (single-false-extra rate ~3× the champion's). The crop-accuracy gains are real; the presence damage is what fails the wild gates. The proposed mechanism — local-head gradients leaking into the shared trunk — was a testable hypothesis, and v13 (part 5) was built to test it. It turned out to be wrong.

The instrument in action, part 5 — the v13 verdict (not promoted, but the attribution the campaign needed). v13 is the controlled experiment the v12 diagnosis called for: the same local-finger head, warm-started from v11 with the HaMeR wild labels, but with the local path learning from detached trunk and backbone features (h.detach() + feature-map detach) — so the presence head and shared trunk receive exactly the champion's gradients, and any presence damage cannot arrive through the gradient path. The verdict is decisive in both directions:

• The tip gains survived full isolation. TIP-PCK@0.05 +0.9pt overall (0.348 → 0.356), near-camera tips +1.4pt @0.05 / +2.7pt @0.10, EASY tips +0.9 / +2.0pt (local_2d/frontier_eval/results_v13_36k.json). With zero gradient flowing into the trunk, the isolation refutes trunk co-adaptation as the mechanism — but the seed controls (§7) later placed gains of this size inside the ±1.9pt run-variance band, so whether they are signal or noise is unresolved.
• Wild stability passed — flicker even improved −39% (0.067/s vs the 0.110 champion baseline) — and the freeze audit came back clean: guard engagement below v11's (63 vs 89 held frames; local_2d/audit/v13_promotion/freeze_audit.json + pair stills). No v12-style guard-manufactured win.
• **And GT-frontier presence still failed — harder than v12. Single-false-extra rate 0.070 vs the champion's 0.006 (specificity 0.924 vs 0.994): on the 158 single-hand GT frames, v13 hallucinates a phantom second hand on 11 frames where v12 had 3 and v11 had 1 — a monotone 1 → 3 → 11** progression across v11 → v12 → v13.

Because the local path is fully gradient-isolated, this result refutes the gradient-leakage diagnosis: the presence damage cannot be flowing through the local head's gradients. Two hypotheses remain standing. (a) Continued-training drift — v12 and v13 are both v11 + 36k extra steps (and v11's own baseline was likewise v7 + 36k), so the phantom inflation may simply be what continued training does to the presence frontier, local head or not. (b) Loss-level coupling — even with detached features, the global heads see errors already absorbed by the residual, coupling the heads at the loss level rather than the gradient level. The decisive control — v14: v11 + 36k continued steps with NO local head, same data and seed — has now run, and it is perfectly clean: single-false-extra 0.0063 and specificity 0.9941 — identical to v11 (with tips at +0.2pt noise-level parity). Continued-training drift is ruled out; the phantom inflation is attributable to the local head, and with gradients excluded (v13), the surviving mechanism is loss-level coupling: the residual heads absorb part of the localization error, changing the gradient that the global heads feed back through the shared trunk — perturbing presence even though the local path itself is detached. The fix is now concrete: score the global heads on pre-residual predictions (trunk trains exactly as the champion) and let the residual learn only the leftover — the design for the next local-head attempt. One more instrument lesson from the episode: COCO calibration does not see this failure at all (v13 ECE 0.053 — v11-level); only the GT-frontier presence block catches it. A gate that watched calibration alone would have called v13 clean.

Verdict: not promoted — GT-frontier presence is a hard gate — but this is the most informative rejection of the campaign: one run isolated the crop win as real, cleared the tracker guards, and killed the leading hypothesis for the presence damage. That is exactly what a controlled experiment is for.

v7 baselines — GT-3D (27 clips, 6,480 GT hand-frames, median / p90):

Full numbers: local_2d/frontier_eval/results_v7.json · local_2d/gt3d_eval/results_ours_v7.json.

The instrument also smoke-tests the camera leg: VGGT ego-camera trajectories score ATE(Sim3) 2.2–3.4 mm on the ego GT windows with strong rotation accuracy (RAS 0.67–0.99); the translation-angle score is low (TAS 0.08–0.16), which we report as-is — TAS is ultra-strict on these short ego paths (median GT path length 12.5 cm), where tiny absolute errors dominate the direction of near-zero translations (local_2d/gt3d_eval/results_camera_vggt.json).

Takeaway — One renderer, one 4-panel standard, one gallery across the whole lineage and 17 clips — and a gate that now has two legs: frozen, hashed GT-2D + GT-3D instruments where tip accuracy is first-class, plus a mandatory wild-stability gate born from the v10 rollback — because v10 proved that lab GT alone can promote a model that is genuinely worse where users actually look. The gauntlet has since passed a good model (v11, the first two-leg champion) and rejected two GT-crop-better ones before promotion (v12 — flicker gate + a freeze audit that exposed a guard-manufactured teleport "win"; v13 — the isolation control that proved the tip gains real yet failed the GT-frontier presence block harder still, part 5 — a failure COCO calibration never saw). Every honest weakness (tips at 0.35, truncated reach at 0.28, one reversed promotion, two pre-promotion rejections) is visible by construction.