Fieldwork V4 benchmark results

This report covers the deterministic V4 localization matrix in research-results/benchmark.json: 9 scenes, 3 severity levels, 20 recorded observations at a six-second cadence, and four algorithms under identical pixels, nominal pose, and INS prior. Truth is retained by the evaluator only.

The report contains 27 matrix cases and 2,300 raw rows, including the learning and ablation experiments. All source hashes matched before and after execution. The three scenario partitions ran concurrently on an Apple M1 Max; individual computeMs values remain meaningful, while the wall-clock duration is not an isolated latency result.

Main findings

Hybrid localization succeeds on 19/20 normal observations at severity 0 and 1, falling to 16/20 at severity 2. It also achieves 19/20 at the two milder rotation levels and 14/20 at the largest yaw change. Under cloud severity 2 it falls to 10/20, and it rejects all low-texture observations. The repetitive scene is intentionally ambiguous: Hybrid accepts 1/20 at severity 0 and 0/20 thereafter, instead of claiming a visually plausible but unsupported fix.

Large INS drift uses deterministic 50 m, 125 m, and 250 m prior perturbations. Hybrid succeeds 15/20, 15/20, and 16/20 respectively, demonstrating global retrieval can recover some cases but is not uniformly robust.

Severity-2 comparison

Each cell reports successful accepted fix count (out of 20) / median error (m) over all hypotheses. A successful fix is accepted and within 10 m; a low median alone does not imply that the estimator accepted the observation.

This is not an overall-winner ranking. It exposes different failure patterns: Feature estimates are precise when they verify, whereas the semantic-only implementation has many structurally plausible but wrong accepts. Concrete severity-2 examples are Feature at normal:s2:t12.000 (28.38 m error at 0.920 confidence) and Semantic Geography at normal:s2:t54.000 (663.35 m at 0.429). Classical also falsely accepts repetitive:s2:t6.000 at 13.89 m and 0.782.

The seasonal exact-frame learning experiment starts with 17/20 successful accepted fixes and ends at 17/20. Mean all-hypothesis position error improves slightly from 2.815 m to 2.767 m; accepted-only mean error improves from 2.067 m to 2.011 m. Six observations entered frozen memory, all six were evaluator-audited correct, and no wrong admission occurred. This is a small accuracy gain with no success-rate gain.

Counterfactuals

The seasonal Hybrid ablations use the same frames and frozen memory. Removing appearance reduces successful accepted fixes from 17/20 to 4/20 and increases mean error to 252.21 m. Removing the prior gives 13/20 successful fixes and 29.06 m mean error. Removing features gives 14/20 successful fixes and one false accept. Removing memory returns the baseline 17/20 result and 2.815 m mean error. These are component removals from the same engine, rather than separately tuned algorithms.

Semantic evidence provides no measured localization benefit in this seasonal ablation: removing it matches the frozen-memory Hybrid result exactly at 17/20 successful fixes and 2.767 m mean all-hypothesis error (median 2.00 m, p95 5.45 m). This experiment does not establish that semantic evidence is generally useless; it establishes that it did not alter the recorded positions or aggregate success. A later event-level audit found two acceptance decisions exchanged, so the semantic branch did affect individual decisions despite zero net gain.

At severity 2, mean compute time is approximately 707 ms for Classical, 1,917 ms for Visual Features, 2,152 ms for Semantic Geography, and 809 ms for Hybrid. These are measurements of the current implementations, not an algorithm-family speed ranking. As RESEARCH.md describes, Hybrid uses inexpensive NCC during its local proposal search before producing its combined certificate, while Feature and Semantic Geography evaluate their structural evidence during pose search.

Calibration and limits

Confidence is explicitly a heuristic evidence score, not a calibrated probability. Seasonal Hybrid ECE is 0.456 before learning and 0.450 after it; the reliability bins are in the JSON and should be read as a calibration gap, not a probability guarantee. The environment is procedural 2.5D imagery, not photogrammetry or a full physical camera model. Low-texture and repetitive cases keep pixels fixed across severity; only the shared deterministic prior uncertainty changes, and their reported domain shift is zero.

Replay requires the seed, configuration, frame hashes, and source manifest in the JSON. The current manifest starts with benchmark 1da32f00, localization a386e8e6, geography 580c7801, and memory 595837c1; the complete SHA-256 values are stored in the artifact.

Deliverables and validation

• Architecture, algorithms, methodology, limits, and computed versus illustrative behavior
• Raw JSON, observation CSV
• Experiment charts, PDF, matrix, ablations
• Benchmark screenshot, candidate evidence inspector, live comparison, mobile view

All 33 automated tests passed. Browser validation exercised all four algorithms, three candidates per event, comparison trajectories, ablations, the complete saved report and mobile layout, with no JavaScript errors. A real benchmark-worker smoke run completed 11 evaluations while 933 main-thread heartbeat ticks demonstrated UI responsiveness. Export validation checked 2,300 CSV records, identical image hashes and priors across methods, and all seven current core source hashes.