Back to the whiteboard
On screen 1 the detector scored 0.818 at IoU 0.50 and 0.341 as a COCO average, and the room was asked what to do about it. Here is what the room said, and here is what each of those answers is actually worth, measured on this data with everything else held fixed.
What the room said
| Instinct | votes |
|---|---|
| Label more images | 0 |
| Use a bigger model | 0 |
| Train for longer | 0 |
| Fix the labels | 0 |
| Measure something else | 0 |
| Narrow what it has to cover | 0 |
Also written down: none recorded
0 votes recorded so far. Go back and add more →
What each answer was worth
The two ablation rows are missing. Run
python scripts/ablations.py to measure “more data” and “a bigger model” directly. Until then this screen shows only the interventions that were measured.| Answer | measured effect | how |
|---|---|---|
| Fix the labels | +36.4 points | Correcting the taxonomy and re-labelling the contested subset moved accuracy from 54.1% to 90.5% on a frozen evaluation set. The model was not touched. |
| Use the right tool for the failure | +0.253 AUC | On the defect family absent from training, a model fitted on good samples only scored 0.886 against the supervised detector's 0.633. It needed no examples of the defect at all. |
| Spend the labelling budget better | 280 labels saved | Uncertainty sampling reached random sampling's final score after 200 labels instead of 480. Same annotators, same budget, different order. |
The whole demonstration in one line
The interventions that moved anything worth moving were fixing what the labels meant, and
noticing that one failure was not a modelling failure at all.
What was actually wrong, in the order we found it
- More data helped, at the thing it fixes. It is on the list first because it is the answer the room gave and it was largely right — about the classes already covered, and about nothing else. An intervention can be effective and still be aimed at the wrong failure.
- The metric flattered. AP@50 of 0.818 and AP@75 of 0.239 are the same model. The gap is localisation, and it traces back to weak ellipse labels, not to capacity.
- The aggregate hid a coverage hole. One texture family scored near zero because it was never in the training set. Averaged in, it produced a number describing nothing.
- The slices were confounded. The brightness slice looked like a lighting finding and was very largely the missing class wearing a hat. Acting on it would have bought lights.
- The labels disagreed with themselves. Two defensible readings of a published rubric agreed at κ = 0.31 if the study is built, and the disagreement was systematic rather than noisy. No model beats its labels.
- The operating point was a business question. Choosing the F1-optimal threshold instead of the cost-optimal one costs real money per batch, and the model is identical.
- One failure needed a different kind of model entirely, not a better one of the same kind.
- The edge story did not match the literature, and we reported what we measured.
- The licence was a design constraint, and the permissive choice was also the better-performing one.
What we would do next, in order
- Write the annotation rubric so that it determines a single reading, and re-run the agreement study until κ clears 0.8 before labelling anything else.
- Cover the missing defect families. One image of a family you have never seen is worth more than a thousand of one you have.
- Run the good-samples-only model alongside the detector permanently, as the net for defect families nobody has thought of yet.
- Set the threshold from the cost ratio, review it when the cost ratio changes, and never from F1.
- Only then consider a larger model.
Honest gaps in this demonstration
- Three DAGM classes, not six. Two for training, one held out. The coverage argument would be sharper with more families and the storyline is unchanged.
- The active-learning learner is a linear probe on frozen features, not the detector. Retraining the detector 120 times does not fit in a demonstration, and the screen says so on its own page.
- The two annotation passes are two stated rule-based readings of one published rubric, not two humans. The screen says that too. Real human passes would be better and are what the live exercise collects.
- Latency was measured on a laptop CPU without VNNI. The ratio would differ on a server or an NPU, and screen 8 says which way.
computed just now The vote tally is read from
data/runtime/whiteboard.json on each request.
precomputed Every measured effect in the right-hand table comes
from a named artefact: scripts/ablations.py, scripts/build_corrosion_study.py,
scripts/train_anomaly.py and scripts/active_learning.py.