HackerWhen one component fails, who can keep working?
One process fails. Must unrelated requests wait? Separate failure propagation from throughput limits, then trace what the operating system and service manager can actually isolate.
Questions in this reading · 19
- Can a service survive a local failure?
- Can healthy work continue when one component fails?
- Who manages a service after startup?
- Where does the kernel's job end?
- How far can one bottleneck spread?
- What must happen before work can run in parallel?
- What does a repeated microbenchmark leave out?
- During an incident, how can a command be distinguished from an observation?
- Did the timed loop actually do the work?
- After a local failure, which promise can still be kept?
- A sound is missing. Who can still work?
- Yielding does not create queue space
- Follow the protection to the thinnest wire
- A running machine and a reachable service
- Why are cars braking after the chicken has gone?
- Where does one borrower’s spending cut show up?
- An introduction is different from sustained spread
- After a leak or a cache loss, what takes the load?
- What converts thermal expansion into a buckling problem?
Can a service survive a local failure?
Compare containing an exceptional condition with isolating a failing process. In each case ask which healthy work can still proceed.
Compare original passages and timestamps · 2
it might throttle because the cpu is overheating but it shouldn't crashwhen a process crashes everything else progresses right so most of the system still worksCan healthy work continue when one component fails?
Process isolation and decoupling services address failure at different scales. Both limit propagation, but shared resources can still carry failure across a boundary.
Compare original passages and timestamps · 2
when a process crashes everything else progresses right so most of the system still worksthe ability to isolate bottlenecks so that they don't spread through the systemWho manages a service after startup?
A service manager responds to change across a system. Isolated processes contain failures inside a running application. Each boundary protects different work.
Compare original passages and timestamps · 2
maintaining services needs to listen to hardware and software changesseparating failures by separating things into different processesWhere does the kernel's job end?
Compare the kernel's responsibilities with the work of keeping services running as hardware and software change. Starting a process is just one part of managing its life.
Compare original passages and timestamps · 2
the operating system for this machine includes linux as a component but it also has to include all the restsystemd as a project ends up complementing the linux kernel by providing all of this user space system layerHow far can one bottleneck spread?
Process isolation contains certain runtime failures. Decoupling services can contain a capacity problem. Identify what is still shared before relying on either boundary.
Compare original passages and timestamps · 2
separating failures by separating things into different processesability to isolate bottlenecks so that they don't spreadWhat must happen before work can run in parallel?
Follow the point where workers have to wait. Adding workers helps only after the serial activity that feeds them can keep up.
Compare original passages and timestamps · 2
increasing the chances of pullaways and parallels, unlike the more orderly methodsthe dispatch process itself became a bottleneck and not necessarily justWhat does a repeated microbenchmark leave out?
Compare the costs visible in an isolated hash-table operation with the effect of workload and cache behavior. Microbenchmarks help answer a narrower question than end-to-end measurements.
Compare original passages and timestamps · 2
But without running the data once, how do you know what
dimensions are inconsequential?when you go to time it you're timing the feature of interest which puts it in cash anywayDuring an incident, how can a command be distinguished from an observation?
Cantrill urges incident responders to ask questions the system can answer. The flight-controls lecture gives a concrete distinction: stick input goes to a computer, while a display can report control-surface deflection. Bring that distinction back to a failing service. Once the console says “recovery command sent,” ask how many workers are actually processing work. A command log and feedback from the controlled object observe different points. Two dashboards reading the same send log still provide only one observation.
Passages and timestamps · 3
debugging is the act of asking questions and answering them not guessing what the answer isthe stick's connected to a computer.rudder deflection surfaces, ailerons, where the horizontal stabilizers are at, how the engines are doing.Did the timed loop actually do the work?
Before reordering a waiting chain around a reported speedup, ask what was timed. Carruth makes the intended work observable to the compiler. Berger shows that even real work can time differently when code and data land at different addresses. Inspect the instructions first, then vary the layouts. Repeating the exact same binary a hundred times can leave a lucky placement entirely unchallenged.
Passages and timestamps · 2
In my case the observable side effect is the benchmarking.layout biases measurementAfter a local failure, which promise can still be kept?
With two working propellers D’Andrea’s controller gives up yaw while retaining control of other motion variables. That spells out degraded operation precisely. Langner’s attack analysis shows the complementary problem: the protection system reads a falsified input, so the monitor shares the compromised feedback path. Continued operation depends on surviving actuators and on whether the evidence for “still within bounds” is independent. List each retained control objective next to its measurement source.
Passages and timestamps · 3
We relinquish control of yaw, but roll, pitch and acceleration can still be controlled with algorithms that exploit this new configuration.with fake input data. And as a matter of fact, this fake input data is actually prerecorded by Stuxnet.The idea is to circumvent a digital safety system.A sound is missing. Who can still work?
At id a bad asset could turn into an obvious placeholder. Other people could keep running the game, while its owner could see what had failed. The slot-app team depended on a store interface shared by its entire catalog. Moving a checkbox could halt publication for every variant. Trace each fault one step downstream. One damages an asset in a scene. The other blocks the only route by which products can ship.
Passages and timestamps · 2
providing a default for a failure case will keep the game running and it will quickly show you what's missinga checkbox gets moved 10 pixels to the right, the entire thing falls apartYielding does not create queue space
Long-running jobs on the Apollo Guidance Computer periodically checked for higher-priority work. That contract established who could run next. The queue still had finite room. During the landing, extra load from the radar let jobs accumulate until executive overflow triggered an alarm and a program restart. When reasoning about overload, write the priority policy and the capacity on separate lines. Even a high-priority job needs a place to enter the queue.
Passages and timestamps · 2
Jobs have to manually check at least every 20 millisecondsqueue of jobs build up, which resulted in executive overflowFollow the protection to the thinnest wire
Draw the route outward from the wall socket and mark where the wire gets thinner. The presenter uses a plug fuse to show how protection can extend to the cord itself. Try the same check on a software limit: circle what it actually protects, then ask whether a downstream component reaches its limit before the guard trips.
Passages and timestamps · 2
before the wire gets overloadedso it protects the length of the wireA running machine and a reachable service
The traveling server remains powered by a UPS, but its public address still depends on a cellular link and a tunnel. Sketch power, transport and routing separately. A spinning fan and an unreachable service can coexist. To check the service promise, make a request from outside the machine.
Passages and timestamps · 2
a UPS system to keep the server up during the movea OpenVPN connection to a server that will route to us the server's public IPWhy are cars braking after the chicken has gone?
The first driver brakes lightly. The next notices late and brakes harder, and a third inherits that disturbance. Follow the first car and it has already recovered; watch the line and a braking wave moves toward its tail. Draw three cars and vary reaction delay and spacing separately. Grey then imagines vehicles maintaining their spacing and coordinating acceleration. Which change merely protects one car, and which reduces the disturbance reaching its follower?
Passages and timestamps · 2
brakes a little harder than necessarySelf-driving cars can just be programmed to stay in the middleWhere does one borrower’s spending cut show up?
Keep two household ledgers in Dalio’s model. When A cuts spending, B loses income and faces the next debt payment with less cash. Burke reconstructs a different transfer in a blackout: once a line trips, power shifts onto the remaining lines. In both diagrams you must follow the arrow one step further, but what travels along it matters. Spending is forgone in the first case, while the load is redirected in the second. Mark what shrinks and what moves before tracing the next failure.
Passages and timestamps · 4
one person's spending is another person's incomea borrower's debts are a lender's assets.above a preset limitcascade power off the overloaded line and onto anotherAn introduction is different from sustained spread
In Grey’s account a jump from animals creates an initial case; a continuing chain needs susceptible hosts. He compares an isolated population with a city receiving new people. Return to Burke’s blackout with that distinction in mind. A relay tripped, but the cascade continued only where connected lines picked up the displaced load. The things transmitted differ. In both examples you can ask what additional condition the next step requires after the first has occurred.
Passages and timestamps · 3
In a small, isolated groupmany chances for infection and even then the new-born plague needs a fertile environmentcascade power off the overloaded line and onto anotherAfter a leak or a cache loss, what takes the load?
In the rotating house, a sensor ring between seals creates an observable intermediate state after a leak. In early Facebook, losing a cache node could push work back onto a particular database. Draw normal load, the first fault, and the component receiving the consequence. Both systems may keep running after one failure. The third box tells you what keeping them running costs.
Passages and timestamps · 4
There's a seal, a sensor ring, and a seal.two seals and two sensors failjust add more machines linearlydifferent Memcache boxes would go down.What converts thermal expansion into a buckling problem?
A free rod can expand; adding a support changes the length over which it can bow sideways. The rail example also needs a neutral temperature to tell when thermal loading is tensile or compressive. A thermal expansion coefficient alone therefore leaves out the reference condition at installation and the lateral supports that remain effective.
Passages and timestamps · 2
if it’s hotter than the neutral temperature, the rail experiences compressive stress.restrain the rod at the centerpoint, effectively halving its length.Try it on your own example
Draw the waiting chain and the failure chain for the same incident. Where do they coincide?