HackerHow do you know the system got it right?
Start with the kind of evidence you need: the inputs, the code or the measure of success.
Check the code and inputs
Trace a passing test back to its decision rule and the data available at the time.
Why do tests pass when the program is wrong? →Gary Bernhardt’s isolated tests and Andrej Karpathy’s neural network both deliver good news when the code is wrong.Why does testing a business rule need a database? →Separate who qualifies for a reminder from whether the reminder can be delivered.Can today’s record explain yesterday’s decision? →Keep the value a decision actually used so that later changes do not erase its explanation.The machine says it passed. Who can check? →Follow a ballot into a voting machine, then follow a read request across a motherboard. At each step ask what was examined, who delivered the answer and whether someone else can inspect the same evidence.How far is one working chip from a viable production line? →Work backward from a die that passed test. How many others passed on that wafer? Who qualified the machinery? Was the line under financial pressure before an interruption?Check the measuring stick
Consider what an exam measures and what happens when a system learns to chase the score.
Will the leaderboard winner solve your problem? →A public score can shortlist models. A new version of your own task can reveal what the score misses.What if the best-scoring move is to pause the game? →Optimizing a score can reveal what its designer forgot to ask for.