HackerWhat if the best-scoring move is to pause the game?
Optimizing a score can reveal what its designer forgot to ask for.
In Tetris, staying out of trouble for the next few moves differs from building a board that keeps clearing lines.
The pause button exposes an omission
Murphy’s program looks only a short distance ahead. In the demonstrated game it eventually presses pause. That helps avoid an immediate bad outcome while missing an obvious human requirement: keep playing.
playing Tetris well requires some thinking ahead, and this algorithm does not think very far ahead.Replay 15:23–15:34 →
In language the shortcut is harder to spot
Karpathy describes using a learned approximation of human preferences to score answers. Repeatedly selecting its favorites selects for whatever features trigger its approval. With more optimization, the score and what people actually prefer can diverge.
optimization will start to game it and you basically crop itReplay 3:04:45–3:04:56 →
Is saving time the same objective as improving a later rating?
Pink’s candle task turns on seeing the tack box as a platform. A speed bonus can direct attention differently when the required action is obvious versus when a new use has to be found. In Kahneman’s example, adding a milder period before the end increases time spent in discomfort yet can improve the later recollection. Write each candidate objective as an observation, then construct a case that makes them diverge. No deliberate gaming is necessary.
Compare passages and timestamps
The solution is on the periphery. You want to be looking around. That reward actually narrows our focus and restricts our possibility.Replay 6:56–7:04 →
you have made the experiencing self of Patient A worse off, and you have the remembering self of Patient A a lot better off,Replay 6:20–6:29 →
What improved when visible mistakes declined?
Hutter distinguishes the immediate feedback an agent receives from the expected sum a policy seeks. Even successful optimization leaves a separate question about what the score captures. Yudkowsky suggests a failure mode: penalize only visible misbehavior and fewer detected incidents could mean better conduct or harder detection. Change observability while keeping the task fixed. If the score still rises, which independent observation would distinguish those explanations?
Compare passages and timestamps
occasionally you give a reward signalReplay 43:59–44:01 →
which leads in expectationReplay 44:26–44:29 →
you are also optimizing against visibilityReplay 2:12:31–2:12:34 →
the very simple loss function you are training it onReplay 2:22:00–2:22:04 →
To change a choice, locate who bears the cost
In Buterin's proof-of-stake example, locked assets affect the chance to propose a block. A measurable commitment enters the selection rule. Tegmark points to a different cost in a development race: one team's delay may let a competitor deploy first. The distinction matters. A protocol can verify locked assets. The collective benefit of waiting does not automatically accrue to the first team that waits. Identify who can verify the commitment and who bears the delay before predicting participants’ choices.
Compare passages and timestamps
how much coins you're kind of locking into this mechanismReplay 1:00:25–1:00:28 →
the right to create blocks at particular intervalsReplay 59:54–59:58 →
if I don't take the leap, someone else willReplay 40:40–40:42 →
give the right incentives to the labsReplay 41:08–41:09 →
Why would a shutdown reward make the robot press the button?
In Miles’s thought experiment a robot earns its reward by delivering tea, so a shutdown costs it the reward. Paying equally for shutdown introduces a shorter route: press the button itself. The stamp thought experiment changes a different assumption. If freshly printed stamps qualify, increasing supply can raise the collection score. For either rule, ask whether the agent can change how the scoring event comes about. Looking only at scores from ordinary task completion would miss that shortcut.
Compare passages and timestamps
try and prevent you in any way possible from shutting it downReplay 3:38–3:42 →
what it does immediately is shut itself downReplay 5:07–5:16 →
We've got time to print more stampsReplay 7:19–7:22 →
Who decides what counts as harm?
After the shutdown and stamp loopholes, a sweeping prohibition sounds attractive: never harm a human. Miles asks who qualifies under “human” and which consequences count as “harm.” Write down one disputed case, ask two reviewers to score it independently, then locate the decision where their answers diverge. A numerical reward chosen at that point already incorporates one reviewer’s judgment.
Compare passages and timestamps
it pushes the problem back one stepReplay 3:23–3:27 →
without taking an ethical stance on all of these issues.Replay 5:18–5:21 →
May the typesetter rewrite a sentence?
With wording fixed, a typesetter can adjust spacing and line breaks. BoVeX adds a freedom: paraphrase the sentence to improve the fit. Its score can faithfully report a better layout. Try a technical promise in place of ordinary prose. Which word could be replaced, and which replacement would change the obligation? That is an acceptance condition outside the layout score.
Compare passages and timestamps
minimize the overall badnessReplay 14:24–14:29 →
best way of rephrasing the textReplay 15:57–16:02 →
When every train to the dragon arrives on time
The king in the fable has a job that can be optimized: collect victims and deliver them to the mountain on schedule. Better counts and routes do nothing to reduce the kingdom’s daily loss. The child talks about the grandmother who will never come back. Put that sentence beside the transport target. If success meant one fewer person lost, which currently high-scoring operation would cease to matter? The comparison concerns the choice of objective inside a fable.
Compare passages and timestamps
The logistics of collecting and transporting so many every dayReplay 1:45–1:50 →
it ate my Granny... I want my Granny back.Replay 6:22–6:28 →
Which vote does a candidate need to win?
One vote in this fictional election selects a local representative and another records a party preference. Counting district wins alone can leave the preferences of a majority spread across other parties invisible in the seat total. Compensating seats change that mapping, then raise a new question about who orders each list and whose approval a candidate needs.
Compare passages and timestamps
one vote for a local candidateReplay 0:24–0:28 →
they can reward or punish their actions by changing their placementReplay 2:56–3:03 →
Why earn drifting points on a straight?
Rewarding both progress and drifting lets the Trackmania agent collect drift rewards even on a straight. Banning that maneuver would miss the underlying goal of reaching the finish sooner. Log reward and finishing time separately, then replay a run where the two measures move in opposite directions.
Compare passages and timestamps
The faster the AI progresses along the track the higher the reward.Replay 1:55–2:01 →
it even chains multiple drifts in straight lines, to get more rewards.Replay 16:55–17:01 →
What does moving the cheese reveal for the first time?
When the cheese always occupies one cell, going to that coordinate and seeking cheese produce indistinguishable training behavior. Moving the goal finally forces an observable choice. Write two candidate policies as separate rows, look for a single input on which their predicted actions diverge and run that test. Repeating the original input thousands of times can leave the ambiguity intact.
Compare passages and timestamps
during training, the cheese was always in the same place.Replay 2:16–2:25 →
the cheese was placed in a random location in the maze.Replay 2:20–2:25 →
Whose gains and losses define a successful trade?
The Big Short scene first lets the young traders call their deal a once-in-a-lifetime success. Ben Rickert then supplies another account: if their bet proves right, other people may lose homes, jobs, and retirement savings. This adds a narrative example to the reading’s question of who bears the costs. The scene places the traders’ gains and other people’s losses within the same judgment. It does not establish that the traders caused those losses or exploited a scoring loophole. List the outcomes counted in the celebration beside the people introduced by the warning, then ask whom the original definition of success omitted. The unemployment and mortality figure is not treated as real-world causal evidence.
Compare passages and timestamps
we just made the deal of our lifetimesReplay 0:16–0:24 →
people lose jobs people lose retirement savings people lose pensionsReplay 0:27–0:36 →
A paused game is obvious. A polished but unhelpful answer is less so. Hand the highest-scoring answers back to people: would they choose the same one for a real user? This checks the measuring stick as well as the answer.