Bench results¶

Labels are the neon-avu-eval silver (agreement between four agentic models, not truth; see Learning). L1 (LOCO) cells are pooled held-out decisions over seven leave-one-card-out folds. cov@5% is the share of items that can be answered before the error rate on the answered set exceeds 5%; flip is the answer change under option reversal (choice) or phrasing swap (yes/no). Every number below names its JSON.

End to end: the whole graph, twice per card (carc-fast at L0), 2026-09-25¶

bench e2e runs the decision graph on the seven fixture cards, two repetitions per planner, and reports agreement rather than asserting determinism: the Jaccard overlap of proposed AVU triples between repetitions, and the recall of the neon-avu-eval consensus terms (proposed by every agentic model, or by at least two). Source: bench/results/2026-09-25/RedHatAI__Qwen3-8B-NVFP4.gateway.e2e.json.

At L0 on carc-fast the graph is far from repeatable (per card 0.0 to 0.67) and recovers little of the agentic consensus; proposals per card were 1 to 9. The gateway planner made it worse and cost 96 minutes of wall time at ~4.7 tokens per second. The planner audit (dataset.ontology_applies) rejected gaz, pato, uo and ro on most cards where the static planner had named them. Specificity opened 23 child groups and replaced no parent. These are the numbers the L2 head and curator labels have to move.

Qwen/Qwen3-8B bf16 on the GB10 workstation (local weights), 2026-09-25¶

The same labels through AnyJev's HFBackend on this host: the full vocabulary is read (no top-20 cap, so no erased labels), and L2 heads read the hidden state of one block. L1 (LOCO) and L2 (LOCO) cells are pooled held-out decisions over seven leave-one-card-out folds. The prompts column reads 0 because the local backend had no request counter in this run (added afterwards; the next run reports prefills).

| task | n | level | acc | ece | brier | cov@5% | cov@10% | flip | n_neg | prompts | ms/dec | | neon_annotate | 98 | L0 | 0.439 | 0.474 | 0.970 | 0.000 | 0.000 | 0.143 | 35 | 0 | 817.1 | | neon_annotate | 98 | L1 (LOCO) | 0.294 | 0.529 | 0.569 | 0.000 | 0.000 | | 5 | 0 | 369.2 | | neon_annotate | 98 | L2 (LOCO) | 0.765 | 0.257 | 0.354 | 0.353 | 0.353 | | 5 | 0 | 196.5 | | neon_annotate | 98 | raw | 0.439 | 0.512 | 1.022 | 0.000 | 0.000 | | 35 | 0 | 398.8 | | neon_aspect | 60 | L0 | 0.533 | 0.456 | 0.922 | 0.000 | 0.000 | 0.150 | 0 | 0 | 1917.5 | | neon_aspect | 60 | L1 (LOCO) | 0.517 | 0.162 | 0.633 | 0.000 | 0.000 | | 0 | 0 | 8175.3 | | neon_aspect | 60 | L2 (LOCO) | 0.567 | 0.175 | 0.552 | 0.383 | 0.417 | | 0 | 0 | 2983.5 | | neon_aspect | 60 | raw | 0.550 | 0.420 | 0.841 | 0.117 | 0.117 | 0.250 | 0 | 0 | 447.3 | | neon_keep_avu | 313 | L0 | 0.949 | 0.031 | 0.085 | 0.994 | 1.000 | 0.022 | 11 | 0 | 845.2 | | neon_keep_avu | 313 | L1 | n/a | | | | | | | | | | neon_keep_avu | 313 | L2 | n/a | | | | | | | | | | neon_keep_avu | 313 | raw | 0.962 | 0.036 | 0.074 | 1.000 | 1.000 | | 11 | 0 | 744.1 | | neon_ontology_fits | 190 | L0 | 0.426 | 0.560 | 1.111 | 0.089 | 0.116 | 0.084 | 114 | 0 | 621.4 | | neon_ontology_fits | 190 | L1 (LOCO) | 0.426 | 0.233 | 0.537 | 0.089 | 0.111 | | 114 | 0 | 5381.1 | | neon_ontology_fits | 190 | L2 (LOCO) | 0.837 | 0.078 | 0.231 | 0.547 | 0.768 | | 114 | 0 | 1906.4 | | neon_ontology_fits | 190 | raw | 0.421 | 0.566 | 1.137 | 0.089 | 0.105 | | 114 | 0 | 216.5 | | neon_ontology_for_column | 59 | L0 | 0.441 | 0.508 | 1.027 | 0.169 | 0.271 | 0.186 | 0 | 0 | 3085.4 | | neon_ontology_for_column | 59 | L1 (LOCO) | 0.441 | 0.141 | 0.732 | 0.102 | 0.102 | | 0 | 0 | 6622.9 | | neon_ontology_for_column | 59 | L2 (LOCO) | 0.712 | 0.154 | 0.391 | 0.458 | 0.678 | | 0 | 0 | 1679.8 | | neon_ontology_for_column | 59 | raw | 0.492 | 0.443 | 0.896 | 0.051 | 0.051 | 0.136 | 0 | 0 | 568.5 | | neon_term_choice26 | 44 | L0 | 0.114 | 0.833 | 1.648 | 0.000 | 0.000 | 0.236 | 28 | 0 | 578.2 | | neon_term_choice26 | 44 | L1 | n/a | | | | | | | | | | neon_term_choice26 | 44 | L2 | n/a | | | | | | | | | | neon_term_choice26 | 44 | raw | 0.114 | 0.789 | 1.621 | 0.000 | 0.000 | | 28 | 0 | 191.4 | | neon_term_fits | 285 | L0 | 0.663 | 0.293 | 0.616 | 0.077 | 0.182 | 0.119 | 199 | 0 | 392.3 | | neon_term_fits | 285 | L1 (LOCO) | 0.656 | 0.079 | 0.422 | 0.049 | 0.077 | | 199 | 0 | 2765.1 | | neon_term_fits | 285 | L2 (LOCO) | 0.765 | 0.058 | 0.327 | 0.088 | 0.396 | | 199 | 0 | 1374.7 | | neon_term_fits | 285 | raw | 0.642 | 0.316 | 0.656 | 0.074 | 0.112 | | 199 | 0 | 199.1 | | neon_value_kind | 278 | L0 | 0.421 | 0.477 | 1.022 | 0.014 | 0.014 | 0.385 | 0 | 0 | 1046.0 | | neon_value_kind | 278 | L1 (LOCO) | 0.428 | 0.088 | 0.669 | 0.007 | 0.007 | | 0 | 0 | 1709.4 | | neon_value_kind | 278 | L2 (LOCO) | 0.576 | 0.069 | 0.565 | 0.029 | 0.040 | | 0 | 0 | 1275.0 | | neon_value_kind | 278 | raw | 0.406 | 0.536 | 1.127 | 0.014 | 0.014 | 0.500 | 0 | 0 | 370.4 |

How to read it (bench/results/2026-09-25/Qwen__Qwen3-8B.hf.json):

• term.fits (285 states, 199 negatives): accuracy 0.642 raw, 0.663 L0, 0.656 L1, 0.765 L2; ECE 0.316, 0.293, 0.079, 0.058; coverage at 5% risk 0.074, 0.077, 0.049, 0.088 and at 10% risk 0.112, 0.182, 0.077, 0.396. L2 meets the 0.10 ECE bar with the best accuracy, but only 25 of 285 items can be answered at 5% risk, so term.fits auto stays null.
• column.ontology_fits (190, 114 negatives) is the one question where L2 changes the picture: accuracy 0.837 and ECE 0.078 with coverage 0.547 at 5% risk and 0.768 at 10%, against 0.426 / 0.233 / 0.089 at L1. The policy_suggestions block cites this cell. Before any threshold is adopted the number needs a second label source: these labels are the agentic models' ontology choices, so the head has learnt their habits, not the truth.
• column.annotate (98, 5 negatives above the weight cut at L1/L2): L2 accuracy 0.765 against 0.294 at L1, but with five negatives the coverage figure (0.353) rests on almost nothing and every fold barely passes the class guard.
• column.aspect (60) and avu.value_kind (278) gain accuracy at L2 (0.567, 0.576) and calibrate (ECE 0.175, 0.069) but stay near zero coverage at 5% risk.
• neon_term_choice26 at raw/L0 scores 0.114 locally against 0.227 / 0.205 on the gateway: reading the full vocabulary does not rescue a 26-way choice with a new key per item; the noul-per-candidate design is the right one for this model.
• avu.keep: 11 negatives, no fold may fit, unchanged.
• Against the gateway at L0 the local bf16 checkpoint scores term.fits 0.663 vs 0.611 and annotate 0.439 vs 0.398, with flip rates 0.119 vs 0.393 on term.fits: the NVFP4 quant and the bf16 weights are not interchangeable, which is why bundles are keyed per model (D5).

carc-fast (RedHatAI/Qwen3-8B-NVFP4) through the CARC gateway, 2026-09-25¶

How to read it (bench/results/2026-09-25/RedHatAI__Qwen3-8B-NVFP4.gateway.json):

• term.fits (285 candidate states, 199 negatives): accuracy 0.502 raw, 0.611 at L0, 0.621 at L1; ECE 0.372, 0.262, 0.072. Calibration meets the pre-registered 0.10 bar at L1, but coverage at 5% risk is 0.007 (two items), so every auto threshold stays null.
• neon_term_choice26 is the control: one choice per column over its candidates with a new key per item, so it can only run raw/L0 and never learns (accuracy 0.23 / 0.21).
• column.aspect (K=8) already lost 19 labels to the gateway's top-20 readout at L0; the wider the choice, the more the gateway erases (DESIGN D11).
• avu.keep has only 11 negatives, so no leave-one-card-out fold is allowed to fit; its raw/L0 numbers reflect a positives-heavy label set, not a calibrated gate.
• In M2 the bench and learn fit disagreed on term.fits ECE (0.072 here, 0.231 in the bundle's manifest). The cause was not the fitting: learn fit averaged per-fold ECE and coverage, which are not linear in the items, while the bench pooled the held-out decisions. Since M3 both share one fitting path and both pool (DESIGN D18).

Synthetic backend (planted biases; numbers are not a claim about any model)¶