A corrected taxonomy, and the measured effect of re-labelling
Screen 3 established that two defensible readings of the published rubric agree only moderately, and that almost all of the disagreement is one state wide and runs in one direction. That is a diagnosis. This screen is the treatment, and the measurement of whether the treatment worked.
Where should the scale be cut? Let the disagreement decide
Collapsing an ordinal scale is only worth doing at a boundary the annotators cannot hold. Asserting a cut and then reporting that agreement improved would be circular, so all three possible binary cuts are measured and the data picks one.
| Cut | Cohen's κ between the passes | raw agreement | reading | note |
|---|---|---|---|---|
| good | fair+poor+severe | 1.0000 | 100% | almost perfect | both readings share the same 0.5% coverage test for 'any corrosion', so they agree on this boundary by construction |
| good+fair | poor+severe | 0.2970 | 59% | fair | |
| good+fair+poor | severe | 0.3755 | 84% | fair |
The best non-degenerate cut is good+fair+poor | severe at κ = 0.376, against κ = 0.310 for the full four-state scale. An honest improvement, and a modest one.
What the re-labelling bought, measured against a frozen yardstick
| Training labels | Accuracy on the frozen yardstick | Δ | what changed |
|---|---|---|---|
| Published four-state taxonomy, trained on pass A | 54.1% | +0.0% | |
| Four-state, 209 contested images re-labelled | 60.2% | +6.1% | re-labelling only |
| Four-state, contested images dropped from training | 64.1% | +10.0% | removal only |
| Corrected taxonomy (good+fair+poor | severe) | 81.0% | +26.8% | taxonomy only |
| Corrected taxonomy + 70 re-labelled | 90.5% | +36.4% | both |
| Corrected taxonomy, contested dropped | 89.6% | +35.5% | both, by removal |
Note the third row, which is the inconvenient one: simply deleting the contested images from training beat re-labelling them (64.1% against 60.2%) on the four-state scale. Ambiguous labels were not merely unhelpful, they were actively harmful, and the cheapest intervention available was to stop training on them. That is not the answer anyone wants, and it is the one the data gives.
The borderline cases themselves
The eight most contested images in the set, with what each reading called them and how the corroded area actually splits across the four states. This is what an ambiguous category boundary looks like when you put it on a wall.
What a corrected taxonomy actually is
Before
Four ordinal states from a published bridge-inspection rubric, applied to images by a rule the rubric does not specify. Adjacent states are contested; one reader systematically escalates; the model inherits both problems and a 0.31 κ ceiling.
After
Two states aligned to the decision the plant actually makes — good+fair+poor | severe — plus a written rule for which reading is intended, plus the contested images escalated to adjudication instead of being guessed at. The category boundary now falls where people can agree, and where the maintenance decision changes.
A footnote on the alternative approach. VisA, the other CC BY 4.0
anomaly set, documents four to nine defect types per object in its paper and ships only
normal and anomaly in its released CSV. Reconstructing that dropped
taxonomy by clustering is a legitimate version of this same exercise. It is not used here
because DAGM already covers detection and the corrosion rubric gives genuine, published,
human ambiguity rather than ambiguity we would have had to manufacture.
precomputed Built by scripts/build_corrosion_study.py at 2026-08-09T19:45:14+00:00. Probe: scikit-learn logistic regression on frozen torchvision ResNet-18 ImageNet embeddings, stratified cross-validation. Dataset: VT Corrosion Condition State, DOI 10.7294/16624663.v2, CC0 1.0. Re-run that script to regenerate it; nothing on this page is hardcoded.