From results to evidence

Testing & evaluation

A passing test, a top score and a successful experiment leave a question open: what does the result actually establish? Follow the code, the task and the measurement back to their assumptions.

12readings · three paths

NEW HERE? START HERE

Why do tests pass when the program is wrong?

Start with a familiar green test. Separate matching an expectation from getting that expectation right.

Start reading ↗

It passed. What could still be wrong?

Ask what was checked, then ask how the checker was checked.

01Why do tests pass when the program is wrong?Gary Bernhardt’s isolated tests and Andrej Karpathy’s neural network both deliver good news when the code is wrong.7 sources · traceable passages 02Why does testing a business rule need a database?Separate who qualifies for a reminder from whether the reminder can be delivered.2 sources · traceable passages 03Can today’s record explain yesterday’s decision?Keep the value a decision actually used so that later changes do not erase its explanation.9 sources · traceable passages 04The machine says it passed. Who can check?Follow a ballot into a voting machine, then follow a read request across a motherboard. At each step ask what was examined, who delivered the answer and whether someone else can inspect the same evidence.22 sources · traceable passages 05How do you turn a one-off failure into a replayable test?Suppose an acknowledged transfer disappears during a leader change. Six engineering talks separate reproducible inputs, shrinkable operations and invariants, then ask whether the fix actually exercises the original failure.6 sources · traceable passages

Does a better score mean a better outcome?

Compare what a task covers with what happens when a system learns to chase its score.

01Will the leaderboard winner solve your problem?A public score can shortlist models. A new version of your own task can reveal what the score misses.12 sources · traceable passages 02What if the best-scoring move is to pause the game?Optimizing a score can reveal what its designer forgot to ask for.17 sources · traceable passages 03How much does a successful demonstration establish?A program gives the expected output. Between “it did” and “it will” lie the test distribution, formal assumptions and deployment context. Those boundaries are easy to miss under pressure to ship.49 sources · traceable passages

Does the conclusion survive a different measurement?

Revisit a good result through experimental conditions, measurement boundaries and use cases.

01Did the experiment measure what the everyday claim promises?Start with a morning walk. Give each observation a precise verb: moved the eyes, reported a letter, measured an enzyme. When the verb changes, the research may be answering a different question.30 sources · traceable passages 02Why does moving the measurement change the verdict on a machine?Open a dishwasher after its first drain, then remove the bread from a toaster. Two small experiments expose where a measurement begins, where it ends and what the instrument actually senses.22 sources · traceable passages 03What use case does a good rating actually measure?Start with an incandescent bulb’s lifetime, then examine detergent, a heat pump, a winter cycle route and a food label. Recover the working conditions behind each advertised measure.13 sources · traceable passages 04How far is one working chip from a viable production line?Work backward from a die that passed test. How many others passed on that wafer? Who qualified the machinery? Was the line under financial pressure before an interruption?8 sources · traceable passages

BEYOND THIS FIELD

Take a question into another field.

These links come from existing readings. They do not imply that two fields share the same standards of proof.

Representation & understanding →

An editorial connection

Familiar questions and real mastery

Doing well under familiar conditions may only mean recognising a pattern seen before. Both readings ask for a test in a new setting, whether the one being tested is a model or a student.

How can you tell a correct answer from understanding? ↗Will the leaderboard winner solve your problem? ↗

Check the passages on both sides

a valid environment, many repetitions, timely feedback, and thousands of hours of deliberate practice.

The Expert Myth · 16:33 ↗

At its core, expertise is recognition.

The Expert Myth · 16:22 ↗

See these connections on the map →

Software & systems →

An editorial connection

A pure interface promises less

Turning a decision into a same-input, same-output computation lets the interface promise only the result, not call order or outside state. That is the other reading’s question: which behaviours should never have become promises?

Why does testing a business rule need a database? ↗What does a module promise its callers? ↗

Check the passages on both sides

a true functional interface, same input, same output.

Rich Hickey · Simple Made Easy · 37:13 ↗

start seeing interconnections between things that could be independent

"Simple Made Easy" - Rich Hickey (2011) · 59:28 ↗

See these connections on the map →

Learning & intelligence →

An editorial connection

Falling loss can hide a bug

The same micrograd walkthrough appears in both. One uses it to show that a falling loss does not prove the code right; the other uses it to explain learning by gradient. Knowing what the gradient computes is how you judge whether “getting better” is real.

Why do tests pass when the program is wrong? ↗How far is next-token prediction from answering a person? ↗

Check the passages on both sides

you may have lots of bugs in the code and uh your network might actually work

Andrej Karpathy · 手写 micrograd · 133:44 ↗

efficiently evaluate the gradient of some kind of a loss function

The spelled-out intro to neural networks and backpropagation: building micrograd · 0:55 ↗

See these connections on the map →