Methods

How we test our own claims.

Every method has to find problems we planted on purpose, and we publish the cases where it doesn't.

Planted-corruption test Finding

A score of 0.964 that still caught nothing where it counted

On our gripper_release corruption, a label-free check reached AUROC 0.964 and recall@13 of 0.000. Why a good-looking score can miss every top pick.

AUROC 0.964 recall@13 0.000

Draft topic

Same dataset, opposite curations

Curate once for throughput and once for reliability, retrain, and see whether each policy wins on its own metric.

Draft topic

Why we state the metric before the result

Four reasonable summaries of one evaluation can name different winners, as PhAIL showed. What that means for data claims.