Ranking the unlabelled pool by uncertainty, against labelling at random

A pool of 1,380 unlabelled images, 180 of them defective. Two strategies buy labels 40 at a time from a starting set of 40: one picks whatever the current model is least sure about, the other picks at random. 5 seeds each, averaged, because a single run of this comparison is noise and would let either side win.

Final, uncertainty sampling

0.459

PR-AUC at 480 labels

Final, random sampling

0.425

same budget

Gap

+0.034

uncertainty sampling wins

Labels saved

280

reached random's final score at 200

The two learning curves

Shaded by nothing — the individual seed runs are drawn faintly so the spread is visible rather than asserted. Uncertainty sampling is ahead early and the margin narrows; that shape is typical and is the honest version of the claim vendors make.

The queue itself

The twelve images the model is least certain about, after 40 labels. This is what an annotator would actually be handed.

#1 · p(defect) 0.50

truth: clean · class 1

#2 · p(defect) 0.50

truth: clean · class 1

#3 · p(defect) 0.50

truth: defective · class 1

#4 · p(defect) 0.49

truth: clean · class 1

#5 · p(defect) 0.49

truth: clean · class 1

#6 · p(defect) 0.49

truth: clean · class 1

#7 · p(defect) 0.49

truth: clean · class 1

#8 · p(defect) 0.51

truth: clean · class 1

#9 · p(defect) 0.49

truth: clean · class 1

#10 · p(defect) 0.51

truth: clean · class 2

#11 · p(defect) 0.51

truth: clean · class 1

#12 · p(defect) 0.48

truth: clean · class 1

8% of this queue is defective, against a pool base rate of 13%. That is worth saying out loud, because it is routinely misunderstood: an uncertainty queue is not a defect-finding queue. It is an ambiguity queue. If you promise the quality team that active learning will surface defects, you have promised the wrong thing, and they will conclude the system does not work.

What the learner is, and what this does not show

Learner: scikit-learn logistic regression (BSD-3-Clause), L-BFGS, class-balanced, on frozen torchvision ResNet-18 ImageNet embeddings (BSD-3-Clause)
Task: image-level: does this part carry a defect
Metric: average precision (area under precision-recall), on the held-out test split
Protocol: 12 rounds × 5 seeds per strategy, evaluated on 576 held-out images

Stated plainly: The learner is a linear probe on frozen features, not the detector. Retraining the detector 120 times would not fit in a demonstration. The pool, the labels, the held-out test set and the uncertainty are real; the curve is about label efficiency, which is what the screen claims.

One further honesty note. An earlier version of this experiment used a hand-rolled gradient-descent logistic regression and reported that random sampling won by 0.084. It had not converged. Swapping in a properly converged L-BFGS solver reversed the result. The lesson is not about active learning at all: an under-fitted learner produces confident, publishable, wrong conclusions about everything downstream of it, and the plateau that looked like a finding about sampling was an artefact of the optimiser.

precomputed Built by scripts/active_learning.py at 2026-08-09T19:36:36+00:00. Pool of 1,380 images, 12 rounds, 5 seeds per strategy. Re-run that script to regenerate it; nothing on this page is hardcoded.