HackerTube English?
软件边界与变化 →

坏掉一个部件,谁还能继续工作?

一个进程出了错,其余请求是否必须等待?把故障传播和性能瓶颈分开看,再追踪操作系统与服务管理者各自能控制的边界。

本文的问题 · 20
  1. 遇到局部异常时保住剩余服务
  2. 隔离故障的目标,是让一处失败不必拖垮仍然健康的工作
  3. 集中管理生命周期,同时限制故障传播范围
  4. 系统管理职责超出 Linux 内核边界
  5. 进程隔离如何变成瓶颈隔离
  6. 并行工作前,先拆掉串行入口
  7. 只测一个反复运行的小例子,可能看不到真实负载的性能成本
  8. 在事故现场,怎样区分命令与观察?
  9. 看似没有瓶颈,是因为工作真的做了吗?
  10. 局部坏了以后,哪一项承诺还能兑现?
  11. 一个音效坏了,谁还可以工作?
  12. 一个会让步的任务,也可能等不到空位
  13. 保险丝保护谁,取决于它在哪一端
  14. 机器还在运行,外边的人还连得上吗?
  15. 一个人少花的钱,会从谁的账上消失?
  16. 只发生一次,与能够一直接下去
  17. 漏一滴水、掉一台缓存,下一道防线在哪里?
  18. 同一根钢轨受热,为什么有时变成长一点,有时向侧面弯?
  19. 重启失败任务时,哪些工作不能一起丢掉?
  20. 故障时保住服务,还是缩小重做范围?

遇到局部异常时保住剩余服务

Hetzner高温测试要求机器可以降频但不崩溃;BEAM演示单个计算进程崩溃后其余请求继续执行。两者都把异常时保持可用的行为当作设计目标,分别在硬件热控制与进程隔离层实现。

对照原文与时间戳 · 2
it might throttle because the cpu is overheating but it shouldn't crash
Over 200,000 Servers in One Place! Visiting Hetzner in Falkenstein (Germany) · ▶ 18:47–19:18
when a process crashes everything else progresses right so most of the system still works
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:12–14:23

硬件降频不是进程故障隔离;对照不意味着两套机制互相替代或具相同保证。

隔离故障的目标,是让一处失败不必拖垮仍然健康的工作

进程隔离与服务间解耦是两个尺度的例子;共享资源仍可能引入级联故障。

对照原文与时间戳 · 2
when a process crashes everything else progresses right so most of the system still works
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:15–14:23
the ability to isolate bottlenecks so that they don't spread through the system
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 9:40–9:49

集中管理生命周期,同时限制故障传播范围

systemd 演讲解释服务管理需要响应硬件和软件变化,已超出一次性启动命令;Erlang/Elixir 演讲展示进程隔离如何让局部失败不拖垮其他活动。并读可以区分系统层生命周期管理与进程层故障遏制这两种互补的可靠性职责。

对照原文与时间戳 · 2
maintaining services needs to listen to hardware and software changes
The Tragedy of systemd · ▶ 12:43–12:55
separating failures by separating things into different processes
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:37–14:43

systemd 与 BEAM 处在不同系统层次;材料没有主张服务管理器本身采用 Erlang 式进程模型。

系统管理职责超出 Linux 内核边界

硬件演讲说明整机的资源与保护职责横跨 Linux 和外围固件;systemd 演讲则把用户态动态服务纳入系统层。两者都要求从实际管理职责出发画系统边界,避免把一个内核当成全部系统。

对照原文与时间戳 · 2
the operating system for this machine includes linux as a component but it also has to include all the rest
USENIX ATC '21/OSDI '21 Joint Keynote Address-It's Time for Operating Systems to Rediscover Hardware · ▶ 17:53–18:07
systemd as a project ends up complementing the linux kernel by providing all of this user space system layer
The Tragedy of systemd · ▶ 15:32–15:45

外围固件与 systemd 不是同一层,也不互相替代;共同点仅是系统职责边界大于 Linux 内核。

进程隔离如何变成瓶颈隔离

Erlang/Elixir 讲解将不同进程作为隔离失败的边界;WhatsApp 的规模化实践则把解耦目标表述为隔离瓶颈,避免局部问题扩散。把两者并读,可以从运行时故障隔离延伸到生产系统的容量与故障域设计。

对照原文与时间戳 · 2
separating failures by separating things into different processes
The Soul of Erlang and Elixir • Sasa Juric • GOTO 2019 · ▶ 14:37–14:43
ability to isolate bottlenecks so that they don't spread
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 9:40–9:48

WhatsApp 的讲解不证明所有瓶颈隔离都由 BEAM 进程隔离实现,也描述了特定系统架构中的取舍。

并行工作前,先拆掉串行入口

后排优先登机使旅客堵在同一放行李位置;WhatsApp 即使有许多工作进程,单个分发进程仍成瓶颈。两者都需把待处理工作分散到多个可同时服务的位置。

对照原文与时间戳 · 2
increasing the chances of pullaways and parallels, unlike the more orderly methods
The Fastest Way to Board a Plane Feels Like the Slowest · ▶ 3:30–3:47
the dispatch process itself became a bottleneck and not necessarily just
Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp · ▶ 12:53–13:02

登机是物理队列,WhatsApp 是进程调度;这里只迁移串行入口限制吞吐的机制,不主张直接套用登机算法。

只测一个反复运行的小例子,可能看不到真实负载的性能成本

两者具体讨论哈希表与缓存。结论是要覆盖使用情境,不是微基准没有用。

对照原文与时间戳 · 2
But without running the data once, how do you know what dimensions are inconsequential?
CppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step” · ▶ 20:19–20:25
when you go to time it you're timing the feature of interest which puts it in cash anyway
Modern Dictionaries by Raymond Hettinger · ▶ 48:33–48:41

在事故现场,怎样区分命令与观察?

Cantrill 建议在故障中先提出能由系统回答的问题。飞控讲解给了一个直观对照:飞行员动了杆,动作先进入计算机;显示器另有控制面偏转信息。把它带回服务故障,控制台显示「恢复命令已发出」时,下一问可以是「工作进程实际回来了几个?」命令记录与被控对象的反馈来自不同位置;如果两者都出自同一条发送日志,第二项检查就没有增加观测。

原文与时间戳 · 3
debugging is the act of asking questions and answering them not guessing what the answer is
Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017 · ▶ 39:50–40:02
the stick's connected to a computer.
Special Lecture: F-22 Flight Controls · ▶ 21:59–22:00
rudder deflection surfaces, ailerons, where the horizontal stabilizers are at, how the engines are doing.
Special Lecture: F-22 Flight Controls · ▶ 31:17–31:25

看似没有瓶颈,是因为工作真的做了吗?

上游一段代码看似变快,先别据此给整个等待链重新排序。Carruth 展示的计时循环要先确保编译器保留想测的工作;Berger 演示即使指令做了工作,函数和对象恰好落在不同地址也能改写结果。两种复核针对不同误差:先检查机器指令,再改变布局复测。只把同一二进制连跑一百遍,很难发现那个二进制恰好抽中了好位置。

原文与时间戳 · 2
In my case the observable side effect is the benchmarking.
CppCon 2015: Chandler Carruth "Tuning C++: Benchmarks, and CPUs, and Compilers! Oh My!" · ▶ 44:30–44:34
layout biases measurement
"Performance Matters" by Emery Berger · ▶ 15:34–15:39

局部坏了以后,哪一项承诺还能兑现?

D’Andrea 的飞行器只剩两个可用旋翼,控制器放弃控制偏航,仍能管住另外几个运动量。这是一份具体的降级约定。Langner 分析的攻击则动了保护系统读取的输入,监控走的是同一条被伪造的反馈路径。一个系统能否带故障运行,不只取决于还剩几个执行部件,也取决于判断「仍在安全范围」的观测是否独立。对自己的设计,可以成对列出保留的控制目标及其测量来源。

原文与时间戳 · 3
We relinquish control of yaw, but roll, pitch and acceleration can still be controlled with algorithms that exploit this new configuration.
The astounding athletic power of quadcopters | Raffaello D'Andrea · ▶ 7:19–7:28
with fake input data. And as a matter of fact, this fake input data is actually prerecorded by Stuxnet.
Ralph Langner: Cracking Stuxnet, a 21st-century cyber weapon · ▶ 7:18–7:25
The idea is to circumvent a digital safety system.
Ralph Langner: Cracking Stuxnet, a 21st-century cyber weapon · ▶ 7:44–7:49

一个音效坏了,谁还可以工作?

id 让缺失的资源换成显眼的占位物,保住其他人继续跑游戏的能力,也让负责资源的人看得出哪里坏了。批量上架老虎机应用的团队面对的是另一个边界。商店调整复选框,后台就可能发不出任何产品。可以把这两个故障分别画成一条箭头:前者损伤一个场景中的一项资产,后者切断所有产品共用的一条发布路径。保护点取决于谁还要继续做什么。

原文与时间戳 · 2
providing a default for a failure case will keep the game running and it will quickly show you what's missing
The Early Days of id Software · ▶ 11:35–11:45
a checkbox gets moved 10 pixels to the right, the entire thing falls apart
1,500 Slot Machines Walk into a Bar: Adventures in Quantity Over Quality · ▶ 31:03–31:08

一个会让步的任务,也可能等不到空位

阿波罗制导计算机要求长任务定期检查是否有更高优先级的工作。这个约定回答了谁先运行,却没有保证工作队列永远装得下。登月时雷达占用额外处理能力,等待的工作积压,触发了 1202 警报和程序重启。分析一次过载时,把调度顺序和队列容量分别写在纸上:给关键工作更高优先级之后,新任务仍要从哪里拿到登记自己的位置?

原文与时间戳 · 2
Jobs have to manually check at least every 20 milliseconds
34C3 - The Ultimate Apollo Guidance Computer Talk · ▶ 36:29–36:32
queue of jobs build up, which resulted in executive overflow
34C3 - The Ultimate Apollo Guidance Computer Talk · ▶ 58:01–58:07

保险丝保护谁,取决于它在哪一端

从墙插往外画,线可能变细,能承受的电流也随之变化。影片用插头中的保险丝说明怎样把保护范围落到延长线上。它给软件中常说的「已有保护」一个很好的检查动作:圈出受保护的那一段,再问下游有没有比触发条件更早达到极限的部件。

原文与时间戳 · 2
before the wire gets overloaded
Perhaps the weakest link in the US electrical system · ▶ 8:53–8:56
so it protects the length of the wire
Perhaps the weakest link in the US electrical system · ▶ 16:23–16:27

机器还在运行,外边的人还连得上吗?

搬上公共交通的服务器仍在电池供电下工作,公网地址还需要通过蜂窝链路和隧道送达。把供电、传输与路由画成三层,就能构造很简单的反例:风扇仍转着,客户端请求却到不了。要验收服务,需要在机器之外发出一次请求。

原文与时间戳 · 2
a UPS system to keep the server up during the move
Moving online webserver using public transport · ▶ 0:45–0:48
a OpenVPN connection to a server that will route to us the server's public IP
Moving online webserver using public transport · ▶ 3:13–3:20

一个人少花的钱,会从谁的账上消失?

Dalio 请你同时做两个家庭的账:甲减少支出,乙就少了一笔收入。轮到乙偿还自己的债务时,甲的决定已经改变了乙的处境。Burke 重建停电时用的是另一种传递:断开一条线,功率转到剩下的线上。两张图的箭头都要追到下一站,但箭头上流动的东西不同。钱没有花出去,负荷却并没有因一条线路打开而消失。沿图排查之前,先写清这个动作让哪笔量减少、哪笔量移去了别处。

原文与时间戳 · 4
one person's spending is another person's income
How The Economic Machine Works by Ray Dalio · ▶ 16:08–16:12
a borrower's debts are a lender's assets.
How The Economic Machine Works by Ray Dalio · ▶ 20:41–20:44
above a preset limit
James Burke Connections, Ep. 1 "The Trigger Effect" · ▶ 11:38–11:44
cascade power off the overloaded line and onto another
James Burke Connections, Ep. 1 "The Trigger Effect" · ▶ 12:03–12:06

只发生一次,与能够一直接下去

在 Grey 的解释里,从动物传给人创造了第一例;要形成更长的传播链,还得不断遇见易感宿主。他拿相对孤立的人群和人口持续进入的城市作比较。也可以带着这一区别回看停电:继电器动作是起点,但只有仍连接的线路不断接过被转移的负荷,故障才继续扩大。两者传播的东西不同,追踪时都可以先问:第一步已经发生之后,下一步还需要什么条件?

原文与时间戳 · 3
In a small, isolated group
Why Some Diseases Only Traveled One Direction · ▶ 2:44–2:52
many chances for infection and even then the new-born plague needs a fertile environment
Why Some Diseases Only Traveled One Direction · ▶ 6:05–6:13
cascade power off the overloaded line and onto another
James Burke Connections, Ep. 1 "The Trigger Effect" · ▶ 12:03–12:06

漏一滴水、掉一台缓存,下一道防线在哪里?

旋转房屋的流体接头在两层密封之间插入传感环;一次泄漏的预期后果,是出现可察觉的中间状态。早期 Facebook 的缓存节点退出后,原本由缓存分担的请求却可能集中落回数据库。给两种设计各画三格:正常负载、首个故障、故障后的接收者。两边都会「继续工作」,但只有看第三格,才知道继续工作的代价由谁承担。

原文与时间戳 · 4
There's a seal, a sensor ring, and a seal.
I thought this rotating house was impossible. · ▶ 5:41–5:44
two seals and two sensors fail
I thought this rotating house was impossible. · ▶ 5:49–5:52
just add more machines linearly
CS50 Lecture by Mark Zuckerberg - 7 December 2005 · ▶ 7:35–7:40
different Memcache boxes would go down.
CS50 Lecture by Mark Zuckerberg - 7 December 2005 · ▶ 12:08–12:11

同一根钢轨受热,为什么有时变成长一点,有时向侧面弯?

一根自由的杆可以伸长;增加中间约束后,要比较的变成多长的一段可以向侧面弯。讲者再引入无应力的中性温度,说明实际钢轨在天气变化时可能受拉或受压。检查安全余量时,仅给出钢材的热胀系数缺少两项条件:铺设时的基准温度和当前仍有效的横向支撑。

原文与时间戳 · 2
if it’s hotter than the neutral temperature, the rail experiences compressive stress.
Why Railroads Don't Need Expansion Joints · ▶ 11:51–12:01
restrain the rod at the centerpoint, effectively halving its length.
Why Railroads Don't Need Expansion Joints · ▶ 10:52–11:03

重启失败任务时,哪些工作不能一起丢掉?

Wills 把恢复描述为一组有边界的选择:可以只重启一个任务,也可以重启更大范围而保留位置和目标等关键信息。他进一步介绍用检查点重新进入计算、让必要的控制工作继续进行,以及借遥测判断系统还在做什么。这个例子提醒我们,恢复机制不仅要说明「怎样再跑一次」,还要说明丢弃什么、信任什么、哪些动作重复会有后果。它扩展了故障传播范围的问题,但不证明所有重启都会成功;还必须检查状态是否有效、故障是否持续以及恢复后的控制是否及时。这里采用的是讲者的设计解释,未核验原代码或飞行记录,也不把其简化的警报故事当作安全认证。

原文与时间戳 · 2
restart just the failed job
Light Years Ahead | The 1969 Apollo Guidance Computer · ▶ 42:21–42:28
leave vital information like where you are and where you're landing intact
Light Years Ahead | The 1969 Apollo Guidance Computer · ▶ 42:28–42:36

故障时保住服务,还是缩小重做范围?

得州电网解说把限电描述为维持供需平衡和系统频率的办法,同时指出一些用户原本应经历轮换停电,却因可调线路耗尽而长时间断电。copyparty 在上传前为每个数据块生成校验和,分块设计也让中断后能够续传,减少从头重传的工作。Joe Armstrong 提醒,副本故障相互独立时,复制可大幅提高可靠性,也主张用小而可验证的单元组合系统。三者处理的层次不同:电网在系统稳定与用户供电之间分配代价,分块上传限制恢复范围,独立副本针对共同故障风险。设计恢复策略时,先写清楚必须保持的服务、允许降级的部分,以及用户承担的恢复成本。

原文与时间戳 · 6
matching supply and demand to maintain a stable frequency
What Really Happened During the Texas Power Grid Outage? · ▶ 3:47–3:52
What should have been rolling outages couldn’t roll
What Really Happened During the Texas Power Grid Outage? · ▶ 14:58–15:02
it generates a list of checksums one for each chunk.
introducing copyparty, the FOSS file server · ▶ 6:08–6:13
the chunked uploading also means that the uploads are resumable.
introducing copyparty, the FOSS file server · ▶ 7:01–7:04
replicate things and keep them independent
"The Mess We're In" by Joe Armstrong · ▶ 18:13–18:19
make them out of small units which we can validate and then and then connect them together
"The Mess We're In" by Joe Armstrong · ▶ 44:05–44:14

读完再想一步

对同一次事故,分别画出等待链和故障传播链。两条线在哪里重合?