System.Xml.XmlElement
Last week’s guest on the podcast was Sam Newman , who had released a book I’ve personally long awaited titled Building Resilient Distributed Systems .
上周播客的嘉宾是 Sam Newman，他发布了一本我个人期待已久的书，名为《构建弹性分布式系统》。
It’s an especially timely book for many of us software engineers, as we’re seeing more outages with nearly all services and systems – much of it caused by more code deploys thanks to AI agents writing more code; and more bugs causing more incidents in production.
对于我们许多软件工程师来说，这是一本特别及时的书，因为我们看到几乎所有服务和系统都出现了更多的中断——其中大部分是由于人工智能代理编写了更多代码而部署了更多代码造成的；更多的错误会导致更多的生产事故。
Sam’s book being “fresh out of the oven,” it naturally also touches on what AI changes as we design resilient systems.
萨姆的书“刚出炉”，它自然也涉及到人工智能在我们设计弹性系统时发生的变化。
In fact, Sam told me that he took a total of three years to finishe the book – instead of the two he originally planned – because he went back and reworked all parts of the book, updating parts where AI is relevant to talk about.
事实上，萨姆告诉我，他总共花了三年时间才完成这本书——而不是他最初计划的两年——因为他回去修改了本书的所有部分，更新了与人工智能相关的部分。
I asked Sam if he would consider sharing an excerpt from the book, with sections especially relevant for anyone building systems that ought to be reliable.
我问 Sam 是否会考虑分享本书的摘录，其中的章节特别适合任何构建可靠系统的人。
Sam graciously agreed, and today’s issue covers a few important topics, spread across two chapters from his book.
山姆慷慨地同意了，今天的问题涵盖了一些重要的主题，分布在他书中的两个章节中。
We get into:
我们进入：
Disaster recovery (DR).
灾难恢复（DR）。
Recovering from catastrophic failures: planning for cloud region/data center/site-wide outages, the concept of RPO and RTO, DRO objectives and SLOs.
从灾难性故障中恢复：规划云区域/数据中心/站点范围的中断、RPO 和 RTO 的概念、DRO 目标和 SLO。
Disaster Recovery Patterns.
灾难恢复模式。
Backup-and-restore, pilot light failover, warm failover and multisite active-active.
备份和恢复、指示灯故障转移、热故障转移和多站点主动-主动。
Infrastructure availability when planning for DR. Ongoing GPU and CPU shortages mean you cannot expect infrastructure availability at all times, so you need to understand constraints your vendors have, and to prioritize critical services.
规划灾难恢复时的基础设施可用性。持续的 GPU 和 CPU 短缺意味着您无法期望基础设施始终可用，因此您需要了解供应商所面临的限制，并确定关键服务的优先级。
Resiliency of AI-powered software.
人工智能驱动的软件的弹性。
Model provider resilience, the challenge of models changing, and the need for observability.
模型提供者的弹性、模型变化的挑战以及可观察性的需求。
How to handle uncertainty.
如何处理不确定性。
Tactics like being multi-model and multi-vendor, making costs transparent, and frequently analyzing vendors.
诸如多模型和多供应商、使成本透明以及经常分析供应商等策略。
Even though this article is pretty long, we cover less than a full chapter from the book's 16 chapters.
尽管本文相当长，但我们只涵盖了本书 16 章中的不到一整章。
If you build reliable systems for a living, I warmly recommend the book itself, and you can get it from the publisher .
如果您以构建可靠的系统为生，我强烈推荐这本书本身，您可以从出版商那里获得它。
My usual disclaimer: as with all my recommendations, I was not paid to recommend this book, and none of the links are affiliate ones.
我通常的免责声明：与我的所有推荐一样，我没有付费推荐这本书，并且没有一个链接是附属链接。
See my ethics statement for more details.
有关更多详细信息，请参阅我的道德声明。
The excerpts below are from Building Resilient Distributed Systems , by Sam Newman.
以下摘录自 Sam Newman 的《构建弹性分布式系统》。
Copyright © 2026 Sam Newman.
版权所有 © 2026 萨姆·纽曼。
Published by O’Reilly Media, Inc. Used with permission.
由 O’Reilly Media, Inc. 出版。经许可使用。
We’ve already discussed how to handle a variety of failures that impact individual service instances.
我们已经讨论了如何处理影响单个服务实例的各种故障。
But what happens when a catastrophic failure causes widespread outage of large parts of your system?
但是，当灾难性故障导致系统大部分内容大范围中断时，会发生什么情况呢？
Fire, flood, electrical supply issues, and errors in the underlying cloud infrastructure are all outside of your control, and at one fell swoop, they can render your attempts to make your system more robust totally redundant.
火灾、洪水、电力供应问题以及底层云基础设施中的错误都超出了您的控制范围，并且一下子就会使您为使系统更加强大而进行的尝试变得完全多余。
Handling these situations falls under the auspices of disaster recovery (DR), a set of procedures that should kick in when something really bad happens.
处理这些情况属于灾难恢复 (DR) 的范围，灾难恢复是在发生真正糟糕的情况时应启动的一组程序。
As with much of what we’ve looked at in this book, trade-offs abound.
正如我们在本书中看到的大部分内容一样，权衡比比皆是。
As you reduce your period of acceptable downtime or data loss, cost and complexity increase, so picking the right approach for your system can be a delicate balancing act.
随着可接受的停机时间或数据丢失的时间减少，成本和复杂性也会增加，因此为您的系统选择正确的方法可能是一个微妙的平衡行为。
In this chapter, I’ll give you some concrete targets to work with to help drive your own DR approach.
在本章中，我将为您提供一些具体目标，以帮助推动您自己的灾难恢复方法。
The exact terminology you use to discuss your infrastructure will likely vary based on implementation details, and this in turn can make discussions around generally applicable patterns more difficult.
用于讨论基础设施的确切术语可能会根据实现细节而有所不同，这反过来又会使围绕普遍适用的模式的讨论变得更加困难。
When running on the public cloud, for example, you’ll probably be familiar with the concept of a cloud region, which is where your cloud resources live.
例如，当在公共云上运行时，您可能会熟悉云区域的概念，这是云资源所在的位置。
When running on a private cloud or in a more traditional on-premises setup, you’ll instead think in terms of data centers.
当在私有云或更传统的本地设置中运行时，您会考虑数据中心。
As we discuss the various options around handling DR, the patterns that we’ll see later in this chapter can be used for either public or private cloud scenarios.
当我们讨论处理灾难恢复的各种选项时，我们将在本章后面看到的模式可用于公共或私有云场景。
Therefore, to keep things simple, I’ll use the term site as a general descriptor of where your computing infrastructure lives.
因此，为了简单起见，我将使用术语“站点”作为计算基础设施所在位置的一般描述。
There are some specific issues to consider around mixing cloud vendors as part of a DR strategy, and we’ll look more at those in Chapter 11.
作为灾难恢复策略的一部分，混合云供应商需要考虑一些具体问题，我们将在第 11 章中详细讨论这些问题。
A spectrum of DR options are available to you, and they can range greatly in terms of cost, complexity, and recovery.
您可以使用一系列灾难恢复选项，它们在成本、复杂性和恢复方面差异很大。
To help you decide which DR approach is most appropriate, we should first try to understand how to determine the appropriate requirements for your context.
为了帮助您确定哪种灾难恢复方法最合适，我们应该首先尝试了解如何确定适合您的环境的要求。
A simple way to define technical requirements around DR is to make use of two concrete measurable targets, recovery point objective and recovery time objective.
定义灾难恢复技术要求的一个简单方法是利用两个具体的可衡量目标：恢复点目标和恢复时间目标。
In Figure 10-1, you can see how these objectives relate to an incident:
在图 10-1 中，您可以看到这些目标如何与事件相关：
The recovery point objective (RPO) defines the maximum amount of time that can have passed since the data was last backed up or synchronized.
恢复点目标 (RPO) 定义自上次备份或同步数据以来可以经过的最长时间。
Put another way, RPO defines the upper duration of potential data loss.
换句话说，RPO 定义了潜在数据丢失的上限持续时间。
If the data for your system backs up once an hour, then it’s possible to lose up to an hour of data if your system has an incident.
如果您的系统数据每小时备份一次，那么当您的系统发生故障时，可能会丢失最多一小时的数据。
Therefore, your RPO would be an hour.
因此，您的 RPO 将为一个小时。
The recovery time objective (RTO) defines how long it’s acceptable for your system to be unavailable in the wake of an incident.
恢复时间目标 (RTO) 定义了系统在发生事件后可接受的不可用时间。
If all your computers became unavailable, perhaps because of a data center–wide outage, how long would it take you to have your services running in an alternative data center?
如果您的所有计算机都变得不可用（可能是由于数据中心范围内的中断），您需要多长时间才能让您的服务在替代数据中心运行？
Your RTO includes all the time needed to recover data from the last recovery point, make computing resources available for your workloads, and redirect traffic to your new site—so there are more variables to consider in terms of bringing your system back online than just time.
您的 RTO 包括从上一个恢复点恢复数据、为您的工作负载提供计算资源以及将流量重定向到新站点所需的所有时间，因此在使系统恢复在线方面需要考虑的变量不仅仅是时间。
If asked what RPO and RTO you’d want in a vacuum, you’d probably give a very simple answer: you’d want an RPO of zero (i.e., a guarantee of no data loss), and an RTO of zero too.
如果被问到您想要什么 RPO 和 RTO，您可能会给出一个非常简单的答案：您希望 RPO 为零（即保证不丢失数据），RTO 也为零。
This is an understandable wish, but as you reduce your RPO and RTO, cost and complexity increase dramatically—and in technical terms, it’s not possible to guarantee a zero RPO or RTO in any case.
这是一个可以理解的愿望，但随着您降低 RPO 和 RTO，成本和复杂性会急剧增加 - 从技术角度来说，在任何情况下都不可能保证零 RPO 或 RTO。
An additional complication is that different types of workloads might require different RPOs and RTOs.
另一个复杂之处是不同类型的工作负载可能需要不同的 RPO 和 RTO。
Consider a bank, for example.
以银行为例。
Having a low RPO to ensure that payment information isn’t lost would be key, and having a low RTO to ensure that people can still take money out of their account is vital.
降低 RPO 以确保支付信息不会丢失是关键，而降低 RTO 以确保人们仍然可以从帐户中提取资金也至关重要。
On the other hand, the part of the system that enables new customers to sign up for an account could have a higher RTO and RPO, as the impact to the business is less severe.
另一方面，系统中允许新客户注册帐户的部分可能具有更高的 RTO 和 RPO，因为对业务的影响不太严重。
So when deciding on the appropriate RPO and RTO, there needs to be a discussion with stakeholders regarding end-user needs and regulatory and legislative requirements, which are balanced against the cost and complexity of implementing the required solution.
因此，在决定适当的 RPO 和 RTO 时，需要与利益相关者就最终用户需求以及监管和立法要求进行讨论，并根据实施所需解决方案的成本和复杂性进行权衡。
For example, your product owner might initially say they need an RPO of 10 minutes, but if you could tell them it would take months of work to implement (during which other work has to be deprioritized), they might reduce their target, perhaps accepting an RPO that is easier for you to deliver.
例如，您的产品所有者最初可能会说他们需要 10 分钟的 RPO，但如果您可以告诉他们需要数月的时间才能实施（在此期间必须降低其他工作的优先级），他们可能会降低目标，也许会接受您更容易交付的 RPO。
Your target RPO and RTO should also exceed anything specified in any service level agreements you have.
您的目标 RPO 和 RTO 还应该超过您拥有的任何服务级别协议中指定的任何内容。
At first glance, RPO and RTO would appear to make for good service level objectives (SLOs).
乍一看，RPO 和 RTO 似乎有助于实现良好的服务水平目标 (SLO)。
After all, they are both important measures that directly impact user experience.
毕竟，它们都是直接影响用户体验的重要措施。
However, there are some challenges around managing RPO and RTO as SLOs.
然而，将 RPO 和 RTO 作为 SLO 进行管理存在一些挑战。
When determining if we are meeting an SLO, we need to collect lots of service level indicators (SLI) to confirm whether or not we are on target.
在确定是否满足 SLO 时，我们需要收集大量服务级别指标 (SLI) 来确认我们是否达到目标。
So if we had an SLO that defined 99th percentile latency over a one-month period of less than 500 milliseconds, we would analyze the latency-related SLIs for that monthlong period to make sure we were meeting the target.
因此，如果我们的 SLO 定义了一个月内小于 500 毫秒的第 99 个百分位延迟，我们将分析该月内与延迟相关的 SLI，以确保我们达到目标。
With RPO and RTO, though, what would our SLI be?
但是，有了 RPO 和 RTO，我们的 SLI 会是什么？
The reality is that we don’t experience disasters regularly enough to gather useful SLIs.
现实情况是，我们并没有经常经历灾难来收集有用的 SLI。
We can of course define related SLOs around things related to our DR approach.
我们当然可以围绕与我们的 DR 方法相关的事物定义相关的 SLO。
For example, it would be possible to define an SLO around the speed of data replication, as this is something we are doing in production and can programmatically measure.
例如，可以围绕数据复制的速度定义 SLO，因为这是我们在生产中所做的事情，并且可以通过编程方式进行测量。
But at best SLOs are only tangentially measuring RPO and RTO.
但 SLO 充其量只是间接衡量 RPO 和 RTO。
So RPOs and RTOs don’t really fit neatly into the SLO concept.
因此，RPO 和 RTO 并不完全符合 SLO 概念。
Of course, that doesn’t mean that you can’t still test your system to make sure it is delivering on your DR objectives—something we’ll discuss more later in the chapter.
当然，这并不意味着您仍然无法测试您的系统以确保它实现您的灾难恢复目标——我们将在本章后面详细讨论这一点。
With the discussion of RPO and RTO in mind, let’s take a look at some common patterns used to implement DR in the wake of a major incident.
考虑到 RPO 和 RTO 的讨论，让我们看一下在重大事件发生后实施灾难恢复的一些常见模式。
In Figure 10-2, we can see an overview of each of the four patterns we will explore, arranged (broadly speaking) in order of increasing complexity and improving RTO and RPO.
在图 10-2 中，我们可以看到我们将要探索的四种模式的概述，它们按照复杂性增加以及 RTO 和 RPO 改进的顺序排列（广义上）。
If starting from scratch, many of these patterns can build on one another, allowing you to incrementally increase the complexity (and cost) of your DR solution over time.
如果从头开始，其中许多模式可以相互构建，从而允许您随着时间的推移逐渐增加灾难恢复解决方案的复杂性（和成本）。
For example, a backup and restore approach requires rerouting, a pilot light setup will add the need for cross-site data replication and infrastructure automation, and a warm failover adds on the cost of having some infrastructure ready to go.
例如，备份和恢复方法需要重新路由，指示灯设置将增加对跨站点数据复制和基础设施自动化的需求，而热故障转移会增加一些基础设施准备就绪的成本。
So if you aren’t sure where to start, working your way through the patterns from left to right in Figure 10-2 makes a lot of sense.
因此，如果您不确定从哪里开始，那么按照图 10-2 中从左到右的模式进行操作会很有意义。
If you have different RPO and RTOs for different parts of your system, you may also end up using a mix of these patterns.
如果系统的不同部分有不同的 RPO 和 RTO，您最终也可能会混合使用这些模式。
Note that they all require some form of rerouting of traffic, and most require some form of cross-site data replication.
请注意，它们都需要某种形式的流量重新路由，并且大多数需要某种形式的跨站点数据复制。
Those options are discussed later in this chapter.
这些选项将在本章后面讨论。
Recovery by restoring data from a backup and loading it into a replacement system.
通过从备份恢复数据并将其加载到替换系统中来进行恢复。
As outlined in Figure 10-3, with this pattern, data is periodically backed up to a separate site.
如图 10-3 所示，使用此模式，数据会定期备份到单独的站点。
In the event that the primary site fails, infrastructure needs to be provided at a new site, and the backed-up data restored.
如果主站点发生故障，需要在新站点提供基础设施，并恢复备份数据。
All data that you want to make available to the failover site needs to be regularly backed up, with the backup data stored somewhere other than the primary site.
您想要提供给故障转移站点的所有数据都需要定期备份，并将备份数据存储在主站点以外的其他位置。
This would ideally be automated, as manual infrastructure provisioning would likely lead to longer RTO and an increased chance of errors.
理想情况下，这应该是自动化的，因为手动基础设施配置可能会导致更长的 RTO 并增加出错的机会。
Data is loaded from the backup location into the newly provisioned infrastructure.
数据从备份位置加载到新配置的基础设施中。
Once infrastructure is available and the data is restored, traffic can be switched to the secondary.
一旦基础设施可用并且数据恢复，流量就可以切换到辅助站点。
Ideally, your failover site runs the same core technology stack as the primary site, meaning that any investment in infrastructure automation on your primary can also be used on the failover (for example, being able to reuse Terraform scripts).
理想情况下，您的故障转移站点运行与主站点相同的核心技术堆栈，这意味着主站点上对基础设施自动化的任何投资也可以用于故障转移（例如，能够重用 Terraform 脚本）。
Keeping the code to automate setup of the primary and the failover the same is very useful, as it means that you’ll have a higher degree of confidence that spinning up the infrastructure on the failover site will work.
保持主站点和故障转移的自动设置代码相同非常有用，因为这意味着您将更有信心在故障转移站点上启动基础设施将发挥作用。
If your failover site uses a different technology stack—meaning it’s not possible to share automated infrastructure provisioning code between the primary and secondary sites—then it becomes even more vital that your DR failover process is regularly tested.
如果您的故障转移站点使用不同的技术堆栈（这意味着无法在主站点和辅助站点之间共享自动化基础设施配置代码），那么定期测试 DR 故障转移过程就变得更加重要。
A caveat to spinning up infrastructure on demand here comes in terms of guaranteeing that infrastructure is actually available when you need it, something we’ll explore later in this chapter.
这里对按需启动基础设施的一个警告是确保基础设施在您需要时实际上可用，我们将在本章后面探讨这一点。
This pattern is generally applicable in situations where your primary site is unavailable and you can tolerate a long RTO, and especially in those situations where you want to keep costs low.
此模式通常适用于主站点不可用且您可以容忍较长 RTO 的情况，特别是在您希望保持较低成本的情况下。
It’s important to note that regular tests of a backup and restore process are necessary to ensure that it actually works.
值得注意的是，有必要定期测试备份和恢复过程，以确保其实际工作。
One approach I’ve found useful is to perform a backup and restore to a secondary site as part of other regular activity (for example, a regular load test exercise).
我发现一种有用的方法是执行备份并恢复到辅助站点，作为其他常规活动（例如，常规负载测试练习）的一部分。
The RPO of this pattern will depend on the frequency of backup.
此模式的 RPO 将取决于备份频率。
The RTO will be the longest of the patterns we’ll look at, and its time will depend primarily on how long it takes to restore data from the backup and provision replacement resources.
RTO 将是我们将要考虑的最长的模式，其时间主要取决于从备份恢复数据和配置替换资源所需的时间。
With a fully automated process and smaller data volumes, this could be in the minutes to hours range.
通过完全自动化的流程和较小的数据量，这可能在几分钟到几小时的范围内。
Manual recovery processes will likely extend these timelines greatly, as will the size of backups involved—terabytes of data take longer to send than gigabytes!
手动恢复过程可能会大大延长这些时间，所涉及的备份大小也会如此——发送 TB 级数据比发送 GB 级数据需要更长的时间！
Another major factor will be availability of the required computing resources.
另一个主要因素是所需计算资源的可用性。
If you end up having to purchase replacement equipment, this pattern could easily result in a timeline of weeks or longer.
如果您最终不得不购买替换设备，这种模式很容易导致需要数周或更长时间的时间。
Data is constantly synced to a failover site, with infrastructure provisioned after failover.
数据不断同步到故障转移站点，并在故障转移后配置基础设施。
With a pilot light setup, data is continually replicated from the primary to the failover site, as shown in Figure 10-4.
通过指示灯设置，数据会不断地从主站点复制到故障转移站点，如图 10-4 所示。
Infrastructure is not provisioned until a failover is required, ensuring that costs can be kept down when working with public cloud infrastructure.
在需要故障转移之前不会配置基础设施，从而确保在使用公共云基础设施时可以降低成本。
This pattern builds on a lot of the work done with a backup and restore, the main change being that the data is continually synchronized to the failover site.
此模式建立在通过备份和恢复完成的大量工作的基础上，主要的变化是数据不断同步到故障转移站点。
As such, the steps involved look fairly similar to the backup and restore pattern:
因此，所涉及的步骤看起来与备份和恢复模式非常相似：
As data is committed to the primary, it is continually replicated to the failover site.
当数据提交到主站点时，它会不断地复制到故障转移站点。
This can be done in a variety of ways, such as by making use of native database technology or application-level replication.
这可以通过多种方式完成，例如利用本机数据库技术或应用程序级复制。
The assumption is that databases and datastores are already present on the failover site, so only remaining resources like compute would need to be provisioned.
假设故障转移站点上已经存在数据库和数据存储，因此只需要配置计算等剩余资源。
Once infrastructure is available and the data is restored, traffic can be switched to the failover.
一旦基础设施可用并且数据恢复，流量就可以切换到故障转移。
From an implementation point of view, the new source of complexity comes from the replication of data from the primary to the failover.
从实施的角度来看，新的复杂性来源来自于将数据从主服务器复制到故障转移服务器。
For systems designed with this process in mind, it might be somewhat trivial to implement.
对于考虑到此过程而设计的系统，实施起来可能有些微不足道。
However, things become more complex when adapting existing systems if you’re making use of a data product that doesn’t support this replication strategy natively.
然而，如果您使用的数据产品本身不支持这种复制策略，那么在调整现有系统时，事情会变得更加复杂。
This is worthy of a more in-depth discussion later in this chapter, especially as the next two patterns also rely on this capability.
这值得在本章后面进行更深入的讨论，特别是因为接下来的两个模式也依赖于此功能。
When compared to a backup and restore, the pilot light failover allows for a much shorter RPO because of more frequent data replication, while also reducing RTO by eliminating the time taken for data restoration.
与备份和恢复相比，由于数据复制更加频繁，指示灯故障转移可以缩短 RPO，同时还可以通过消除数据恢复所需的时间来减少 RTO。
The cost-saving of a pilot light failover really makes sense only in the context of the public cloud.
指示灯故障转移的成本节省只有在公共云环境中才真正有意义。
Costs are reduced primarily in terms of not paying for infrastructure unless it’s used.
成本的降低主要是因为除非使用基础设施，否则无需支付基础设施费用。
But if your failover site is your own private infrastructure that you have to pay for regardless of whether it is used, then some form of a warm failover makes a lot more sense.
但是，如果您的故障转移站点是您自己的私有基础设施，无论是否使用您都必须付费，那么某种形式的热故障转移就更有意义。
At that point you’re already paying for the hardware, so you might as well use it.
那时你已经为硬件付费了，所以你不妨使用它。
Data is synced to a failover site, with at least some infrastructure already provisioned to handle traffic.
数据同步到故障转移站点，至少已经配置了一些基础设施来处理流量。
With a warm failover, as shown in Figure 10-5, data is continually replicated from the primary to the failover site, with at least some infrastructure already provisioned in advance of the failover itself.
通过热故障转移，如图 10-5 所示，数据会不断地从主站点复制到故障转移站点，并且至少在故障转移之前已经配置了一些基础设施。
Having some infrastructure provisioned even before the failover occurs results in a reduction in RTO for some functionality.
即使在故障转移发生之前配置一些基础设施也会导致某些功能的 RTO 减少。
How much infrastructure is provisioned on the failover can vary, but the idea is that some capacity already exists to allow for a much faster failover.
在故障转移上配置的基础设施数量可能会有所不同，但其想法是已经存在一些容量以允许更快的故障转移。
A common approach would be to prioritize the subset infrastructure that is provided on the secondary site for the more critical capabilities of the system, allowing for a shorter RTO for the most important functionality.
一种常见的方法是优先考虑辅助站点上提供的基础设施子集，以实现系统更关键的功能，从而为最重要的功能提供更短的 RTO。
Cold, Warm, and Hot Failovers: A brief note is warranted here regarding the term warm.
冷、暖和热故障转移：这里有必要对术语“暖”进行简要说明。
You’ll find various solutions and vendor products promising cold, warm, or hot failover, with the only real distinguishing factor between these terms often being that a hot failover would give you a lower RTO in comparison to a warm failover, and a warm failover would give you a lower RTO than a cold one.
您会发现各种解决方案和供应商产品承诺冷、温或热故障转移，这些术语之间唯一真正的区别因素通常是，与热故障转移相比，热故障转移将为您提供更低的 RTO，而热故障转移将为您提供比冷故障转移更低的 RTO。
Given that what constitutes hot or warm from vendor to vendor and from tech stack to tech stack varies so greatly, I just decided to pick warm and stick with it.
考虑到不同供应商以及不同技术堆栈的“热”或“暖”的构成差异如此之大，我决定选择“暖”并坚持下去。
Naming things is hard.
给事物命名是很困难的。
The “How it works” and “When to use it sections” are cut off from this excerpt, but are present inside the book.
“它是如何工作的”和“何时使用它”部分被从这段摘录中删除，但仍然存在于书中。
The system is distributed across two or more sites, with writes being able to be sent to any site.
该系统分布在两个或多个站点上，写入可以发送到任何站点。
With a multisite active-active setup, data is replicated constantly between two or more sites, as you can see in Figure 10-6.
通过多站点主动-主动设置，数据会在两个或多个站点之间不断复制，如图 10-6 所示。
Traffic for any customer can be served from any site at any time.
可以随时从任何站点为任何客户提供流量。
This pattern sidesteps the need for a failover, as it is assumed that traffic for any user can switch between sites at any time.
这种模式避免了故障转移的需要，因为它假设任何用户的流量都可以随时在站点之间切换。
As such, it delivers the lowest RTO and RPO of any pattern we’ve looked at so far, though at much higher cost and complexity.
因此，它提供了迄今为止我们见过的所有模式中最低的 RTO 和 RPO，但成本和复杂性要高得多。
The “How it works” and “When to use it sections” are cut off from this excerpt, but are present inside the book.
“它是如何工作的”和“何时使用它”部分被从这段摘录中删除，但仍然存在于书中。
At the time of writing, GenAI is being used by more and more industries.
截至撰写本文时，GenAI 正在被越来越多的行业使用。
Whatever your views on the usefulness of this technology (or its broader societal impact), the nature of GenAI workloads for both training and inference purposes has driven ever-increasing demand for infrastructure, in turn leading to reduced capacity for many users.
无论您对这项技术的实用性（或更广泛的社会影响）有何看法，GenAI 训练和推理工作负载的性质已经推动了对基础设施的需求不断增加，进而导致许多用户的容量减少。
Even if you aren’t running what you consider to be specialized GenAI workloads (for example, those requiring specific GPUs), you may well see the impact of these shortages more broadly.
即使您没有运行您认为的专用 GenAI 工作负载（例如，需要特定 GPU 的工作负载），您也可能会更广泛地看到这些短缺的影响。
Aside from the ever-growing cost of GPUs, memory costs have spiked significantly in 2025 and 2026, impacting general server availability.
除了 GPU 成本不断增长之外，内存成本在 2025 年和 2026 年也大幅飙升，影响了服务器的总体可用性。
I’ve already spoken to clients who are struggling to run “normal” (non-AI) workloads on some of the large public cloud providers, as they can’t secure enough capacity.
我已经与一些客户交谈过，他们正在努力在一些大型公共云提供商上运行“正常”（非人工智能）工作负载，因为他们无法确保足够的容量。
Being able to shift regions in some cases is helping, but guarantees of future availability seem fairly weak.
在某些情况下能够转移区域是有帮助的，但对未来可用性的保证似乎相当薄弱。
Even buying your own hardware doesn’t seem to be a quick fix, with lead times for servers seemingly extending further and further with each passing week.
即使购买自己的硬件似乎也不是一个快速解决方案，服务器的交货时间似乎每周都在延长。
This has to be taken into account when considering the patterns detailed previously.
在考虑前面详述的模式时必须考虑到这一点。
If your chosen approach assumes you can dynamically provision infrastructure when a failure occurs, you will have to look at what guarantees you have around the capacity you want.
如果您选择的方法假设您可以在发生故障时动态配置基础设施，那么您将必须考虑您对所需容量的保证。
In a scenario where a major cloud vendor’s region has failed, you may find yourself fighting over scraps with everyone else.
在主要云供应商的区域出现故障的情况下，您可能会发现自己与其他人争夺碎片。
There are three ways to handle this scenario: reserve capacity; prioritize critical services; or lastly, consider running a fully redundant set of infrastructure at all times.
There are three ways to handle this scenario: reserve capacity;优先考虑关键服务； or lastly, consider running a fully redundant set of infrastructure at all times.
Not every vendor offers the ability to reserve capacity, and even then it is limited to certain cloud services.
并非每个供应商都提供预留容量的能力，即使如此，它也仅限于某些云服务。
On AWS, for example, you can pay a fee to guarantee the availability of EC2 instances (managed virtual machines) when you need them.
例如，在 AWS 上，您可以付费以保证 EC2 实例（托管虚拟机）在您需要时可用。
This has a cost impact, as you pay a fee for the reservation whether or not you use the instances (the silver lining being that if you did end up using them, the cost would be cheaper than normal in-demand pricing).
这会产生成本影响，因为无论您是否使用这些实例，您都需要支付预订费用（一线希望是，如果您最终使用它们，则成本将比正常的按需定价便宜）。
This doesn’t help you, though, when using cloud resources for which reservations aren’t possible—you can’t purchase guaranteed availability of Lambda invocations, for example.
不过，当您使用无法预订的云资源时，这对您没有帮助——例如，您无法购买 Lambda 调用的保证可用性。
Next, consider prioritizing critical services.
接下来，考虑确定关键服务的优先级。
If there is a cost to guaranteeing availability (or if only a limited amount of infrastructure is available), then it’s vital that this goes to workloads that have a higher criticality.
如果保证可用性存在成本（或者只有有限数量的基础设施可用），那么将其转移到具有更高关键性的工作负载就至关重要。
This is where having separate RTOs and RPOs on a service-by-service basis comes into play.
这就是基于服务的单独 RTO 和 RPO 发挥作用的地方。
Finally, if you could justify the complexity and expense of having a fully redundant set of infrastructure ready to run your system, then you’d have the confidence that your computing resources will be available, with the caveat that this option can be extremely costly.
最后，如果您能够证明拥有一套完全冗余的基础设施来准备运行您的系统的复杂性和费用是合理的，那么您就有信心您的计算资源将可用，但需要注意的是，此选项可能会非常昂贵。
But that cost can be mitigated if you can make alternative use of the redundant resources.
但如果您可以替代使用冗余资源，则可以降低成本。
Back in Chapter 4, I mentioned BankCo, where we ran calculations across a large pool of machines.
回到第四章，我提到了 BankCo，我们在大量机器上运行计算。
A large portion of these computers were desktop PCs sitting idle in an office situated outside London.
这些计算机中的很大一部分是闲置在伦敦郊外办公室中的台式电脑。
These PCs formed part of the bank’s DR strategy: in the event that traders had to evacuate the bank’s main location, they could relocate to the secondary site and continue working.
这些电脑构成了银行灾难恢复策略的一部分：如果交易员必须撤离银行的主要地点，他们可以搬迁到辅助地点并继续工作。
It was a requirement that these standby PCs were constantly running to ensure that we knew that they worked, and so that updates could be pushed out to them.
要求这些备用电脑不断运行，以确保我们知道它们可以工作，以便可以将更新推送给它们。
This was obviously back in the day when most of the critical work relied on thick-client applications, so they couldn’t be accessed via the web.
显然，当时大多数关键工作都依赖于胖客户端应用程序，因此无法通过网络访问它们。
We were able to make use of these DR workstations to grow our pool of available workers.
我们能够利用这些灾难恢复工作站来扩大我们的可用员工队伍。
This meant that when the DR site wasn’t in operation (which was most of the time), we were able to carry out our calculations much faster, as we could parallelize the work over more machines.
这意味着当灾难恢复站点不运行时（大多数时候），我们能够更快地执行计算，因为我们可以在更多机器上并行工作。
If the DR site’s machines were unavailable to us, we still had a set of our own servers on which we could run the calculations to suit our normal targets.
如果我们无法使用灾难恢复站点的计算机，我们仍然拥有一组自己的服务器，可以在上面运行计算以满足我们的正常目标。
So if you are thinking about securing ring-fenced infrastructure as part of a DR approach, see if you can think of useful ways to put it to good use when it’s not needed for production workloads.
因此，如果您正在考虑将保护环网基础设施作为灾难恢复方法的一部分，请看看是否可以想出有用的方法，在生产工作负载不需要它时充分利用它。
You could use the spare infrastructure for a load test environment, for example; the load test could easily be aborted when DR is needed.
例如，您可以将备用基础设施用于负载测试环境；当需要灾难恢复时，负载测试很容易中止。
We’ve looked at how AI is changing how we build and operate our software.
我们研究了人工智能如何改变我们构建和操作软件的方式。
But we are also integrating AI into the products we build, using it to hopefully improve our software for our users.
但我们也将人工智能集成到我们构建的产品中，希望利用它来为用户改进我们的软件。
What new challenges does this create for ensuring our systems are resilient?
这为确保我们的系统具有弹性带来了哪些新挑战？
Foundation model providers OpenAI and Anthropic don’t have a great track record in terms of providing high levels of availability for their services, at least when compared to other managed service providers, like public cloud vendors.
基础模型提供商 OpenAI 和 Anthropic 在为其服务提供高水平可用性方面没有良好的记录，至少与公共云供应商等其他托管服务提供商相比是这样。
While I’m sure this will improve over time, having a potentially flaky third party in the mix of your critical functionality can be an issue.
虽然我确信这会随着时间的推移而得到改善，但在您的关键功能组合中加入潜在的不稳定第三方可能会是一个问题。
A solution here is to consider ensuring that your software can still operate even if the model is unavailable.
这里的一个解决方案是考虑确保即使模型不可用，您的软件仍然可以运行。
This could be achieved by limiting functionality in the case of the model being unavailable (or you getting rate limited or running out of tokens!)—this is an example of the graceful degradation pattern, which we’ll look at more in Chapter 11.
这可以通过在模型不可用的情况下限制功能来实现（或者你受到速率限制或用完代币！）——这是优雅降级模式的一个例子，我们将在第 11 章中详细介绍。
Another option is to have backup vendors for a given model to hedge against your normal model provider having availability issues.
另一种选择是让给定模型的备用供应商来对冲正常模型提供商出现可用性问题的情况。
For example, you can use OpenAI models on Azure, a handy alternative if accessing the models via OpenAI’s own services proves to be too unstable, and in a similar way Anthropic’s models can also be accessed via Google’s Gemini Enterprise Agent Platform.
例如，您可以在 Azure 上使用 OpenAI 模型，如果通过 OpenAI 自己的服务访问模型被证明太不稳定，这是一个方便的替代方案，并且以类似的方式，也可以通过 Google 的 Gemini Enterprise Agent Platform 访问 Anthropic 的模型。
When using backup vendors like this, there may be a lag in terms of how quickly the latest models may be available, but given that you may not always want to be on the bleeding edge of the models you use to embed in your software (something we’ll discuss in just a moment), this is probably fine.
当使用像这样的备份供应商时，最新模型的可用速度可能会存在滞后，但考虑到您可能并不总是希望处于用于嵌入软件中的模型的最前沿（我们稍后将讨论这一点），这可能没问题。
Consistency of behavior plays a big role in helping us ensure that our system is behaving appropriately.
行为的一致性在帮助我们确保系统正常运行方面发挥着重要作用。
While dealing with the unexpected is a key concept in resiliency (or “graceful extensibility,” as David Woods put it), that doesn’t mean we want to build software where surprise is constant.
虽然处理意外情况是弹性（或大卫·伍兹所说的“优雅的可扩展性”）的一个关键概念，但这并不意味着我们想要构建不断出现意外的软件。
Quite aside from the fact that LLMs are nondeterministic, small changes in models can also lead to quite different behavior.
除了法学硕士具有不确定性这一事实之外，模型的微小变化也可能导致截然不同的行为。
I’ve seen reports where a model has started behaving differently, without any major change in behavior being flagged by the vendor.
我见过一些报告，其中模型开始表现不同，但供应商没有标记出任何重大的行为变化。
Additionally, moving to a newer version of a model can fundamentally change how your prompts may be handled.
此外，迁移到模型的新版本可以从根本上改变处理提示的方式。
It is therefore vital to have a good set of tests to validate that the AI portions of your code are behaving appropriately—and this should be a mix of traditional functional tests, together with the use of AI evaluation tools (evals).
因此，拥有一套良好的测试来验证代码的 AI 部分是否正常运行至关重要，这应该是传统功能测试与 AI 评估工具 (evals) 的使用的结合。
You want to make sure that when the unexpected occurs, you are ones who get surprised first, rather than your users.
您要确保当意外发生时，首先感到惊讶的是您，而不是您的用户。
I can recommend the second edition of Gayathri Mohan’s Full Stack Testing,3 for those of you wanting to explore testing in general, including verification of AI-based systems.
对于那些想要探索一般测试（包括基于人工智能的系统的验证）的人，我可以推荐 Gayathri Mohan 的《全栈测试》第二版3。
For a deeper dive specifically into evals, I’d suggest Evals for AI Engineers by Shreya Shankar and Hamel Husain.
为了更深入地研究评估，我建议您阅读 Shreya Shankar 和 Hamel Husain 编写的 Evals for AI Engineers。
The telemetry we receive from our own software and from third-party services alike is key to making sure our applications are working the way that we expect.
我们从自己的软件和第三方服务等收到的遥测数据是确保我们的应用程序按我们期望的方式运行的关键。
As we discussed back in Chapter 3, we prefer a small variation of latency rather than a broad variation of latency.
正如我们在第 3 章中讨论的那样，我们更喜欢延迟的小变化，而不是延迟的大变化。
If the time it takes to successfully complete an operation varies wildly, it can be difficult to know where we should start triggering timeouts, for example.
例如，如果成功完成操作所需的时间差异很大，则可能很难知道我们应该从哪里开始触发超时。
It also becomes more difficult to define latency-related SLOs.
定义与延迟相关的 SLO 也变得更加困难。
Once you send a prompt to an AI model or instruct an agent to do something for you, the latency for the operation can vary greatly, and in ways that can be difficult to predict from the outset.
一旦您向人工智能模型发送提示或指示代理为您做某事，操作的延迟可能会有很大差异，并且从一开始就很难预测。
Some of this can be put down to variability in the performance of the provider at a given point in time, but it can also be the result of rate limiting by the provider, or your prompt could just be taking longer to process.
其中一些可以归因于提供商在给定时间点的性能变化，但也可能是提供商限制速率的结果，或者您的提示可能需要更长的时间来处理。
In situations where the prompts sent to models come directly from your users, then you will likely have even less certainty about how long it should take for a model to come back to you.
如果发送给模型的提示直接来自用户，那么您可能更不确定模型需要多长时间才能返回给您。
Has a call hung, or is it expecting a large output?
呼叫是否挂起，或者是否需要大量输出？
Some techniques to solve this can include dynamically setting timeouts based on the number of tokens being generated, or monitoring the streaming output from a model so you can track when the response starts coming back rather than waiting for the full response.
解决此问题的一些技术包括根据生成的令牌数量动态设置超时，或监视模型的流输出，以便您可以跟踪响应何时开始返回，而不是等待完整响应。
In the previous sections, I’ve outlined why the world of AI is so uncertain right now.
在前面的章节中，我概述了为什么人工智能世界现在如此不确定。
However, I want this book to focus on practical, actionable advice.
然而，我希望本书重点关注实用的、可操作的建议。
With that in mind, I want to outline some key steps you can take to make sure that you’re prepared for the uncertainty ahead.
考虑到这一点，我想概述您可以采取的一些关键步骤，以确保您为未来的不确定性做好准备。
With model behavior, speed, and quality changing from version to version, it’s vital that you build your software to support changing models.
随着模型行为、速度和质量随着版本的不同而变化，构建软件来支持不断变化的模型至关重要。
Not only do you want the ability to change models to make use of the latest and greatest, but if costs start to increase from one vendor, you want to make sure the barrier to finding another vendor isn’t too great.
您不仅希望能够更改模型以利用最新和最好的产品，而且如果某个供应商的成本开始增加，您还希望确保寻找另一家供应商的障碍不会太大。
It is worth highlighting that mixing models from different vendors in a single flow can exacerbate issues.
值得强调的是，在单个流程中混合来自不同供应商的模型可能会加剧问题。
For example, a flow that involved models from Anthropic, OpenAI, and Google’s Gemini would be impacted if any of those models had availability issues.
例如，如果其中任何模型存在可用性问题，涉及 Anthropic、OpenAI 和 Google Gemini 模型的流程都会受到影响。
All those models have to be available and functioning correctly at the time the flow is executed.
所有这些模型都必须在流程执行时可用且正常运行。
Also, prompts that work against one model may not be entirely portable to another—a prompt tuned against one model will have to be reevaluated when moving to a different model, even if you stay within the same vendor ecosystem.
此外，适用于一种模型的提示可能无法完全移植到另一种模型 - 即使您处于同一供应商生态系统中，在转移到不同模型时也必须重新评估针对一种模型调整的提示。
Taking a multimodel approach extends to your vendors too.
采用多模型方法也适用于您的供应商。
If a vendor you are dependent on is in turn dependent on a single vendor’s model or API, you and your vendor become vulnerable to volatility in that model or to being impacted by a sudden price increase.
如果您所依赖的供应商又依赖于单个供应商的模型或 API，那么您和您的供应商很容易受到该模型波动的影响或受到价格突然上涨的影响。
Cursor was forced to pass on a significant cost increase to its customers in 2025, as it relied on the use of Anthropic’s services, and Anthropic changed its pricing structure, which led to Cursor having to pay much more.
Cursor 被迫在 2025 年将大幅增加的成本转嫁给客户，因为它依赖于使用 Anthropic 的服务，而 Anthropic 改变了其定价结构，导致 Cursor 不得不支付更多费用。
Coincidentally, Anthropic’s Claude Code is a direct competitor of Cursor.
巧合的是，Anthropic 的 Claude Code 是 Cursor 的直接竞争对手。
If Cursor had been able to use AI models from another firm, it might have been less vulnerable to this issue.
如果 Cursor 能够使用另一家公司的人工智能模型，它可能不会那么容易受到这个问题的影响。
Keep Your Options Open: Make sure you aren’t wholly dependent on a single model or AI platform.
保持你的选择余地：确保你不完全依赖单一模型或人工智能平台。
If model quality degrades, or prices from OpenAI or Anthropic go up, you want to make sure that you have the flexibility to go elsewhere.
如果模型质量下降，或者 OpenAI 或 Anthropic 的价格上涨，您需要确保能够灵活地转向其他地方。
Given concerns about rising costs from vendors, and given that pay-as-you-go pricing models can lead to surprising and unsustainable bills, having the ability to see on a regular basis how much money you are spending, and where, can be really important.
考虑到供应商成本上升的担忧，以及即用即付的定价模式可能会导致令人惊讶且不可持续的账单，因此能够定期查看您花费了多少钱以及在哪里花费，非常重要。
This doesn’t need to be about limiting spend necessarily; it can be about understanding how you spend money and understanding what value that brings.
这不一定是限制支出；它可以是关于了解你如何花钱以及了解它带来的价值。
James Stanier, CTO at digital healthcare firm Nordhealth, has advocated for being able to see spend on a per-person basis.
数字医疗公司 Nordhealth 的首席技术官 James Stanier 主张能够查看每个人的支出情况。
Rather than this being about shaming people who spend too much money, he makes the case that this visibility drives better understanding of how the software is being used and helps drive the adoption of tools that are proving to be valuable.
他并不是为了羞辱那些花太多钱的人，而是认为这种可见性可以促进人们更好地了解软件的使用方式，并有助于推动被证明有价值的工具的采用。
In his 2025 talk at LeadDev Berlin, “LLMs: An Operator’s View”, Stanier explains that if you see a person spending a lot of money, this shouldn’t be seen as a bad thing, as typically it showsomeone trying to get more done.
在 2025 年柏林 LeadDev 演讲“法学硕士：运营商的观点”中，斯坦尼尔解释说，如果你看到一个人花了很多钱，这不应该被视为一件坏事，因为通常这表明有人试图完成更多工作。
He cites an example of one of his QA engineers blowing through €1,000 in a single week on Claude Code, which was a much higher spend than normal.
他举了一个例子，他的一位 QA 工程师在 Claude Code 上一周就花费了 1,000 欧元，这比正常情况要高得多。
After pairing with the engineer in question to understand where this money was going, Stanier was able to see that this spend allowed the engineer in question to more than double his productivity.
在与相关工程师合作了解这笔钱的去向后，斯坦尼尔发现这笔支出使相关工程师的生产力提高了一倍以上。
Having the information about spend available to you, and being able to talk to people about how they use these tools and the benefits (or problems) they cause, will help you all make much better and more informed decisions.
掌握有关支出的信息，并能够与人们讨论他们如何使用这些工具以及它们带来的好处（或问题），将帮助你们做出更好、更明智的决策。
It can also help you spot egregious overspend.
它还可以帮助您发现严重的超支情况。
Uber recently revealed that it had burned through its entire annual AI budget in just the first four months of the year, and Microsoft announced it was pulling Claude licenses from its employees, due in part to cost, apparently.5 These stories might cause one or two raised eyebrows, but they pale in comparison to the recent (admittedly unsubstantiated) report from Axios that an unnamed enterprise company spent $500 million on Claude licenses after failing to put cost controls into place.
Uber 最近透露，仅在今年的前四个月就耗尽了其全部年度 AI 预算，而微软则宣布将取消其员工的 Claude 许可证，显然，部分原因是成本。 5 这些故事可能会引起一两个人的惊讶，但与 Axios 最近的（诚然未经证实的）报告相比，它们相形见绌。该报告称，一家未具名的企业公司在未能实施成本控制后，在 Claude 许可证上花费了 5 亿美元。
Tip : Make your AI spend visible!
提示：让您的人工智能支出可见！
It can help you understand how people are using it and also help you spot when the bills start climbing precipitously.
它可以帮助您了解人们如何使用它，还可以帮助您发现账单何时开始急剧攀升。
Vendor management is a common activity that many firms carry out.
供应商管理是许多公司开展的一项常见活动。
It typically consists of a regular review of the main vendors that the organization makes use of to help it make better decisions in the future.
它通常包括对组织利用的主要供应商的定期审查，以帮助其在未来做出更好的决策。
For example, seeing that a large amount of money is being spent with a particular vendor despite the quality of service from said vendor degrading might cause an escalation to either get the price reduced or get the quality of service improved—or it might kick off a process to find a replacement.
例如，尽管特定供应商的服务质量下降，但看到该供应商花费了大量资金，可能会导致升级，要么降低价格，要么提高服务质量，或者可能启动寻找替代供应商的流程。
In my last role at Thoughtworks, I helped lead our internal technology operations.
在我在 Thoughtworks 的最后一个职位中，我帮助领导了我们的内部技术运营。
Although my main job was to help provide support to internal delivery teams, I was also involved from time to time with our own vendor management activities.
虽然我的主要工作是帮助为内部交付团队提供支持，但我也不时参与我们自己的供应商管理活动。
As part of these activities, we would regularly assess the general health of the vendors themselves.
作为这些活动的一部分，我们将定期评估供应商本身的总体健康状况。
During the last review I was involved with, we were actively concerned about the health of one of our main technology suppliers; the supplier was in a very competitive market, and its market share was dropping.
在我参与的上次审查期间，我们积极关注我们主要技术供应商之一的健康状况；该供应商所处的市场竞争非常激烈，其市场份额正在下降。
Although we were happy with the quality of service it provided (and with the cost), our fear was that wider market forces would drive it out of business.
尽管我们对它提供的服务质量（以及成本）感到满意，但我们担心更广泛的市场力量会使其停业。
As a result, we decided to get to know the vendor much better but to also prepare a list of alternative solutions that we could select if the vendor in question didn’t survive.
因此，我们决定更好地了解该供应商，同时准备一份替代解决方案列表，以便在该供应商无法生存时我们可以选择。
Given the degree of uncertainty around the AI sector in general, I’d recommend regular vendor management activities.
考虑到人工智能行业总体的不确定性，我建议定期开展供应商管理活动。
If you already have a normal vendor analysis cycle, consider increasing its frequency for AI vendors.
如果您已经有正常的供应商分析周期，请考虑增加人工智能供应商的频率。
If possible, find out if any vendors are in turn dependent on another provider, as in the case of Cursor being dependent on Anthropic’s cloud services.
如果可能的话，查明是否有任何供应商反过来依赖于另一个提供商，就像 Cursor 依赖于 Anthropic 的云服务的情况一样。
I’d be more concerned about a vendor that is in turn wholly dependent on a single company than I would be about a vendor that has more flexibility about whose services it uses.
我更关心的是完全依赖于单一公司的供应商，而不是在使用其服务方面具有更大灵活性的供应商。
If you’ve never done vendor management before, consider instituting a lightweight process so you can build up a better understanding of how exposed you are to risk in this space.
如果您以前从未进行过供应商管理，请考虑建立一个轻量级流程，以便您可以更好地了解您在该领域面临的风险有多大。
With the concerns around the potential security vulnerabilities that come from the use of AI, I’d also recommend keeping your threat model up-to-date with regular threat assessments.
由于担心使用人工智能带来的潜在安全漏洞，我还建议通过定期威胁评估来保持威胁模型最新。
As with vendor reviews, consider increasing the frequency of your normal threat assessments in areas with heavy use of AI, as this technology and how we use it is changing very quickly.
与供应商审查一样，考虑在大量使用人工智能的领域增加正常威胁评估的频率，因为这项技术以及我们使用它的方式正在迅速变化。
Review Your AI Vendors Regularly!
定期审查您的人工智能供应商！
Given the volatility in the space, make sure you review your AI vendors much more frequently than normal vendors.
考虑到该领域的波动性，请确保您比普通供应商更频繁地审查您的人工智能供应商。
I realize that this chapter may have ended on somewhat of a negative note.
我意识到本章可能会以某种消极的方式结束。
But I wanted to be clear that we have real, practical use cases for AI in helping us build more resilient distributed systems.
但我想澄清的是，我们有真实、实用的人工智能用例来帮助我们构建更具弹性的分布式系统。
Most of the use cases I outlined in this chapter are being used daily by people all over the world.
我在本章中概述的大多数用例都被世界各地的人们每天使用。
The key thing is that the solutions I have shared here still put humans at the heart of decision making.
关键是我在这里分享的解决方案仍然将人类置于决策的核心。
It’s clear that, due to the challenges I have covered here as well, we are a long way from a truly autonomous AI SRE.
很明显，由于我在这里也提到了挑战，我们距离真正自主的 AI SRE 还有很长的路要走。
Despite the obvious benefits of using AI, we do need to be fully aware of the downsides at both the micro and macro level.
尽管使用人工智能有明显的好处，但我们确实需要充分意识到微观和宏观层面的缺点。
At the micro level, LLM-powered AI requires us to be OK with nondeterminism, whereas there are many activities we perform in the context of building resilient systems that need to be fully repeatable and deterministic.
在微观层面上，法学硕士驱动的人工智能要求我们接受不确定性，而我们在构建弹性系统的背景下执行的许多活动需要完全可重复和确定性。
At the macro level, concerns about the AI sector as a whole mean we are dealing with an ever-changing (and hopefully) improving set of products built on very shaky financial foundations.
在宏观层面上，对整个人工智能行业的担忧意味着我们正在应对一组不断变化（并且希望）不断改进的产品，这些产品建立在非常不稳定的财务基础上。
It doesn’t matter how good the tooling is if it’s too expensive to use (or the company making it goes bust).
如果工具的使用成本太高（或者制造工具的公司破产），那么工具再好也无济于事。
As is the nature of things, this uncertainty around AI’s role in our industry and society as a whole will fade as we learn more and as the market settles down.
正如事物的本质一样，随着我们了解更多以及市场稳定下来，人工智能在我们行业和整个社会中的作用的这种不确定性将会消失。
But be warned: I think we are still very much in the middle of the storm, and the calm waters may be several years away yet.
但请注意：我认为我们仍处于风暴之中，平静的水面可能还需要几年的时间。
A lot of this chapter was about uncertainty, and one thing that can make the resiliency of a system quite uncertain is what happens when machines fail.
本章的大部分内容都是关于不确定性的，而使系统的弹性变得相当不确定的一件事是当机器发生故障时会发生什么。
So in Chapter 8, I’ll explore how to make sure our systems still work even when machines let us down.
因此，在第 8 章中，我将探讨如何确保我们的系统即使在机器让我们失望的情况下仍能正常工作。
Gergely, again.
又是格尔盖伊。
Thanks to Sam for sharing this excerpt of the book!
感谢 Sam 分享这本书的摘录！
While this was a long article in itself, we covered less than one of 16 chapters in the book.
虽然这篇文章本身很长，但我们只讨论了本书 16 章中的不到一章。
I found the book an excellent read, and recommend getting it if you’re building production systems:
我发现这本书非常值得一读，如果您正在构建生产系统，建议您阅读它：
My sense is that AI is making “resiliency basics” more important than before.
我的感觉是，人工智能使“弹性基础”比以前更加重要。
Most of the book is not about AI – in fact, AI is barely mentioned in many of the chapters.
本书的大部分内容都与人工智能无关——事实上，许多章节中几乎没有提到人工智能。
But as AI agents generate more code and allow building software faster than before: we now have more software (or more complex software!) to operate.
但随着人工智能代理生成更多的代码并允许比以前更快地构建软件：我们现在有更多的软件（或更复杂的软件！）来运行。
And while there are AI tools and vendors that promise operating software to be easier with AI: I don’t really buy that it is, at least not at scale.
尽管有些人工智能工具和供应商承诺使用人工智能来操作软件会更容易：但我并不真正相信这一点，至少在规模上不相信。
Having the right monitoring and alerting in place is becoming more important, as is responding to outages, and being aware of tricky situations like thundering herds, retries, idempotency, timeouts, and many others.
拥有正确的监控和警报变得越来越重要，响应中断以及意识到棘手的情况（如雷群、重试、幂等性、超时等）也变得越来越重要。
Preparing for disaster recovery seems like a practical thing to do when your service is important enough.
当您的服务足够重要时，为灾难恢复做准备似乎是一件很实际的事情。
I wanted to include excerpts on disaster recovery, because I observe plenty of “disasters” (that is, widespread outages) happening across the industry where recovery time becomes surprisingly long.
我想摘录有关灾难恢复的内容，因为我观察到整个行业发生了大量“灾难”（即大范围的中断），其中恢复时间变得惊人地长。
And some of this could be because the team had not prepared for, or exercised, disaster recovery beforehand.
其中一些可能是因为团队没有事先准备或练习灾难恢复。
A good example is how Coinbase’s global trading service went offline after a single AWS zone had problems.
一个很好的例子是，Coinbase 的全球交易服务在单个 AWS 区域出现问题后就离线了。
As I summarized the incident in The Pulse , back in June:
正如我六月份在《脉搏》中总结的那样：
“Having no automated failovers is incredibly amateurish for an operation of Coinbase’s scale.
“对于 Coinbase 这样规模的运营来说，没有自动故障转移是极其业余的。
Coinbase moves about 5.2 trillion dollars per year, and is valued at around $40B.
Coinbase 每年的交易量约为 5.2 万亿美元，估值约为 $40B。
The outage interrupted around $7 billion-worth of financial activity, based on my napkin math.”
根据我的餐巾纸计算，这次中断中断了价值约 70 亿美元的金融活动。”
It’s smart to learn from the mistakes of others, when we can: and the Coinbase incident felt like one of these.
如果可以的话，从别人的错误中吸取教训是明智之举：Coinbase 事件就是其中之一。
I hope you found these topics relevant, and thanks again to Sam for writing the book, and sharing relevant parts with us.
我希望您发现这些主题相关，并再次感谢 Sam 撰写这本书并与我们分享相关部分。