Advancing human control of military AI
John L.
约翰·L。
Thornton China Center Strobe Talbott Center for Security, Strategy, and Technology
桑顿中国中心 Strobe Talbott 安全、战略和技术中心
The following pieces were authored by members of the U.S. and Chinese delegations who participate in an ongoing Track II Dialogue on Artificial Intelligence and National Security convened by the Brookings Institution’s Foreign Policy program and Tsinghua University’s Center for International Security and Strategy, with support from the Minderoo Foundation.
以下文章由参与正在进行的人工智能与国家安全二轨对话的美国和中国代表团成员撰写，该对话由布鲁金斯学会外交政策项目和清华大学国际安全与战略中心召集，并得到了明德鲁基金会的支持。
The dialogue, which has been running since 2019, aims to identify practical steps to manage risks relating to artificial intelligence (AI) and national security.
该对话自 2019 年以来一直在进行，旨在确定管理人工智能 (AI) 和国家安全相关风险的实际步骤。
The dialogue also undertakes to develop a living glossary of key terms in AI and national security to ensure precision in communication.
该对话还致力于开发人工智能和国家安全关键术语的动态术语表，以确保沟通的准确性。
Below, two authors provide a U.S. and Chinese perspective on steps the United States and China could take to build boundaries around acceptable uses of AI in national security.
下面，两位作者从美国和中国的角度阐述了美国和中国可以采取哪些步骤来围绕人工智能在国家安全中的可接受用途建立界限。
These recommendations, which reflect the authors’ views, are presented in anticipation of a forthcoming U.S.-China government-level dialogue on AI.
这些建议反映了作者的观点，是为即将举行的中美政府级人工智能对话而提出的。
Together, they explore how the United States and China could build on their existing commitment to maintain human control over nuclear weapons as AI becomes increasingly integrated into the military domain.
他们共同探讨了随着人工智能日益融入军事领域，美国和中国如何在现有承诺的基础上继续维持人类对核武器的控制。
The authors examine potential areas for further agreement, including ensuring human control over AI-enabled cyberattacks, as well as practical mechanisms and safeguards that could help translate shared principles into practice.
作者研究了进一步达成协议的潜在领域，包括确保人类对人工智能网络攻击的控制，以及有助于将共同原则转化为实践的实用机制和保障措施。
The United States and China have not developed a common approach to governing AI in the military domain.
美国和中国尚未制定出共同的方法来管理军事领域的人工智能。
This does not mean, however, that they are uninterested in addressing the possibility that some military applications of AI will harm civilians, endanger societies, and undermine global stability.
然而，这并不意味着他们对解决人工智能的某些军事应用可能伤害平民、危害社会和破坏全球稳定的可能性不感兴趣。
In late 2024, the presidents of the United States and China agreed that humans, not AI, should have control over the decision to use nuclear weapons.
2024 年末，美国和中国总统一致认为，应该由人类而不是人工智能来控制使用核武器的决定。
This agreement reflects a shared understanding of the catastrophic consequences of nuclear war, and of the need to minimize the likelihood that it will occur not because leaders have chosen it, but because AI fails or is misused.
该协议反映了对核战争灾难性后果的共同理解，以及需要尽量减少核战争发生的可能性，这不是因为领导人选择了核战争，而是因为人工智能失败或被滥用。
As important as that agreement is, it is only the starting point: As militaries integrate agentic AI into their operations and AI models advance rapidly in sophistication and capability, new pathways to catastrophic outcomes are emerging.
尽管该协议很重要，但这只是一个起点：随着军队将代理人工智能整合到其行动中，并且人工智能模型在复杂性和能力方面迅速进步，导致灾难性后果的新途径正在出现。
Presidents Donald Trump and Xi Jinping can lead in governing these dangers by agreeing that only humans, not AI, should make the decision to initiate a cyberattack against each other’s nuclear command, control, and communications systems (NC3) or critical infrastructure.
唐纳德·特朗普总统和习近平总统可以通过同意只有人类而不是人工智能才能决定对彼此的核指挥、控制和通信系统（NC3）或关键基础设施发起网络攻击来领导治理这些危险。
States rely on NC3 systems to detect threats, make decisions about nuclear use, and transmit secure, reliable launch orders to the forces that would carry them out.
各国依靠 NC3 系统来检测威胁，做出有关核使用的决定，并向执行这些命令的部队传送安全、可靠的发射命令。
Nuclear doctrine does not eliminate these risks.
核理论并不能消除这些风险。
Whether a state relies on no-first-use, launch-on-warning, or other principles, failures in these functions could: cause breakdowns in leaders’ ability to communicate or issue commands; make a state unable to detect or accurately assess threats; cause misperceptions that incentivize a first strike; or result in mistaken or unauthorized nuclear launch.
无论一个国家是否依赖不首先使用、警告后发射或其他原则，这些功能的失败都可能： 导致领导人沟通或发布命令的能力崩溃；使国家无法检测或准确评估威胁；造成误解，从而激发第一次打击；或导致错误或未经授权的核发射。
An intentional AI-powered cyberattack on another state’s NC3 would therefore initiate a military crisis with possibly catastrophic effects.
因此，对另一个国家的 NC3 进行故意的人工智能网络攻击将引发军事危机，并可能造成灾难性影响。
But an intentional attack is not the only way AI-powered cyber capabilities can cause a military crisis.
但故意攻击并不是人工智能驱动的网络能力导致军事危机的唯一方式。
Such capabilities could also initiate an unintended military crisis in two other ways.
这种能力还可能以另外两种方式引发意外的军事危机。
First, even if the state’s objective is only to pre-position an offensive cyber capability, an unplanned attack could still result from human error, technical limitations in the AI models, or AI agents that act in ways that humans did not anticipate.
首先，即使国家的目标只是预先部署进攻性网络能力，意外攻击仍然可能因人为错误、人工智能模型的技术限制或人工智能代理以人类无法预料的方式行事而导致。
Second, a bad actor could use the AI-powered cyber capability to execute an unauthorized attack.
其次，不良行为者可以利用人工智能驱动的网络功能来执行未经授权的攻击。
In both cases, the attacked state would be highly likely to respond by elevating alert levels and readying its nuclear forces.
在这两种情况下，受攻击的国家很可能会通过提高警戒级别并准备核力量来做出反应。
If the disruption to normal NC3 functions were severe enough, leaders would have to make decisions about nuclear employment under conditions of extreme time pressure and deep uncertainty, which might result in nuclear launch and war.
如果NC3正常功能受到的干扰足够严重，领导人将不得不在极端时间压力和高度不确定性的条件下就核使用做出决定，这可能会导致核发射和战争。
AI-powered cyberattacks on critical infrastructure—the economic, financial, health, transportation, and safety systems that support civic life—could also result in an extremely serious military crisis.
对关键基础设施（支持公民生活的经济、金融、健康、交通和安全系统）的人工智能网络攻击也可能导致极其严重的军事危机。
Such an attack could cause the immediate destruction of property, contamination of the local environment, social chaos, and widespread illness, injury, or death.
此类袭击可能会导致财产立即遭到破坏、当地环境受到污染、社会混乱以及大范围的疾病、伤害或死亡。
Like attacks on NC3, attacks on critical infrastructure could be initiated purposefully, or as a result of technical error or unauthorized action.
与针对 NC3 的攻击一样，针对关键基础设施的攻击可能是有意发起的，也可能是由于技术错误或未经授权的操作而发起的。
A state that suffers an attack on critical infrastructure would be highly likely to retaliate militarily, initiating a war with unpredictable escalation dynamics.
关键基础设施遭受攻击的国家很可能会采取军事报复，从而引发一场不可预测的升级动态的战争。
The upcoming U.S.-China AI dialogues are an opportunity for the world’s leading AI countries to meaningfully advance governance of these extremely dangerous uses of AI-powered offensive cyber capabilities.
即将举行的中美人工智能对话为世界领先的人工智能国家提供了一个机会，可以有意义地推进对人工智能驱动的进攻性网络能力的这些极其危险的用途的治理。
Planned government-to-government dialogues can prepare Trump and Xi to agree that decisions to launch cyberattacks on NC3 and critical infrastructure should be made by humans, not AI.
计划中的政府间对话可以让特朗普和习近平同意，对 NC3 和关键基础设施发动网络攻击的决定应该由人类而不是人工智能做出。
Extending the principle applied to nuclear weapons—that attacks with strategic effects should express human intent and must be subject to human accountability—does not require Washington and Beijing to share other principles of cybersecurity or infrastructure protection.
将适用于核武器的原则——具有战略效果的攻击应表达人类意图并必须承担人类责任——并不需要华盛顿和北京分享网络安全或基础设施保护的其他原则。
It is confined to agreement on the narrow point that both sides have the responsibility to minimize the likelihood that attacks with strategic effects will be caused by chance, error, advanced AI capable of exceeding human-defined constraints, or unauthorized actors.
仅限于在狭隘的一点上达成一致，即双方都有责任最大限度地减少偶然、错误、能够超越人类定义限制的先进人工智能或未经授权的行为者造成具有战略效果的攻击的可能性。
Leader-level statements are the necessary precondition for bilateral dialogue on verifiable protocols and safeguards.
领导人声明是就可核查议定书和保障措施开展双边对话的必要前提。
They are also important public signals that the United States and China recognize the special responsibilities that come with leading in AI and are committed to taking pragmatic steps to reduce the likelihood that their military uses of AI will put nations at existential risk.
它们也是重要的公开信号，表明美国和中国认识到人工智能领域的领先地位所带来的特殊责任，并致力于采取务实措施，减少人工智能的军事使用使国家面临生存风险的可能性。
No other signal carries the same weight in demonstrating that both governments understand what is at stake—not just for their own societies, but for everyone.
没有任何其他信号具有同等分量，可以证明两国政府都了解所面临的危险——不仅是为了各自的社会，而且是为了每个人。
AI is penetrating every corner of the military domain at an unprecedented pace.
人工智能正以前所未有的速度渗透到军事领域的各个角落。
From cyber defense to weapon systems, from intelligence analysis to command and decisionmaking, AI’s presence is ubiquitous.
从网络防御到武器系统，从情报分析到指挥决策，人工智能的存在无处不在。
However, the widespread adoption of this technology also brings with it unprecedented risks.
然而，这项技术的广泛采用也带来了前所未有的风险。
In 2024, the leaders of China and the United States reached a consensus on ensuring that nuclear weapons remain under human control at all times.
2024年，中美领导人就确保核武器始终处于人类控制之下达成共识。
In May of this year, the two countries’ heads of state further proposed building a constructive strategic stability relationship .
今年5月，两国元首进一步提出构建建设性战略稳定关系。
The governance of military AI should be an integral part of this effort.
军事人工智能的治理应该是这项工作的一个组成部分。
Yet these political declarations still fall short of preventing AI from spiraling out of control, being misused, or triggering escalations in conflict.
然而，这些政治宣言仍然不足以防止人工智能失控、被滥用或引发冲突升级。
China and the United States should explore how to translate their principled consensus into actionable institutional arrangements, while also identifying the practical challenges currently facing them.
中美双方应探讨如何将原则共识转化为可操作的制度安排，同时也要认清当前面临的现实挑战。
In future dialogues on military AI, broader issues—including the regulation of lethal autonomous weapons systems—should also be incorporated.
在未来关于军事人工智能的对话中，还应该纳入更广泛的问题，包括致命自主武器系统的监管。
Both China and the United States have repeatedly emphasized the importance of ensuring the principle of “human control,” as reflected in their existing joint statements as well as in their respective policy positions.
中美两国都一再强调确保“人为控制”原则的重要性，这体现在两国现有的联合声明和各自的政策立场中。
Melanie Sisson’s article further argues that the authority to launch cyberattacks against nuclear command systems and critical infrastructure must remain firmly in human hands.
梅兰妮·西森的文章进一步指出，对核指挥系统和关键基础设施发起网络攻击的权力必须牢牢掌握在人类手中。
However, to prevent these declarations from becoming mere formalities, China and the United States must establish practical cooperation mechanisms in three key areas.
然而，为了防止这些声明流于形式，中美必须在三个关键领域建立务实的合作机制。
First, they should develop a list of red lines for military AI—clearly prohibiting AI from autonomously launching attacks on NC3 systems or autonomously deciding to use nuclear weapons.
首先，他们应该制定一份军事人工智能红线清单——明确禁止人工智能自主对NC3系统发起攻击或自主决定使用核武器。
At the same time, they should also explore a list of critical infrastructure sectors (such as finance, health care, energy, etc.) and clearly define the boundaries.
同时，还应探索关键基础设施领域（如金融、医疗、能源等）清单并明确边界。
Second, they should reach a consensus on what constitutes “meaningful human control.” In the 1970s, United States Air Force Colonel John Boyd proposed the renowned OODA loop—in which whoever can “observe, orient, decide, and act” more quickly in a competition or conflict will seize the initiative.
其次，他们应该就什么是“有意义的人类控制”达成共识。 20世纪70年代，美国空军上校约翰·博伊德提出了著名的OODA循环——谁能在竞争或冲突中更快地“观察、引导、决策和行动”，谁就能占据主动。
Later, discussions centered on “human-in-the-loop” and “human control” are based on that theoretical foundation.
后来围绕“人在环”和“人控制”的讨论就是基于这个理论基础。
Yet even before Boyd, Chinese Premier Zhou Enlai had already issued instructions demanding “absolute certainty” (万无一失) in China’s nuclear testing efforts.
然而，早在博伊德之前，中国总理周恩来就已经发出指示，要求中国核试验工作“万无一失”。
In the view of Chinese experts and practitioners, this instruction continues to hold sway today and embodies the principle of “human control.” Given the two sides’ fundamentally different institutional frameworks and strategic cultures, even when addressing the same issue, they may employ different terminology and adopt divergent approaches.
在中国专家和实践者看来，这一指示至今仍然具有影响力，体现了“人控”原则。由于双方根本不同的制度框架和战略文化，即使解决同一问题，也可能使用不同的术语、采取不同的做法。
China and the United States need to ensure mutual acceptance of each other’s definitions of “meaningful human control” and related safeguard mechanisms through regular dialogue or by establishing a terminology working group.
中美双方需要通过定期对话或建立术语工作组，确保相互接受对方对“有意义的人类控制”的定义和相关保障机制。
Third, they should establish a China-U.S. military hotline for AI.
第三，建立中美关系。人工智能军事热线。
Building on existing China-U.S. military communication mechanisms—such as the defense minister hotline and the Maritime Military Security Consultation Mechanism—adds a dedicated channel for reporting emergency incidents triggered by AI errors or AI cyberattacks.
以现有的中美关系为基础国防部长热线、海上军事安全磋商机制等军事沟通机制增加了专门的渠道，用于报告人工智能错误或人工智能网络攻击引发的紧急事件。
The goal is to reduce the risk of cascading crises caused by AI malfunctions that could mislead humans or even override human judgment.
其目标是降低人工智能故障引发连锁危机的风险，因为人工智能故障可能会误导人类，甚至超越人类的判断。
The 2023 balloon incident should be a wake-up call for both China and the United States.
2023年的气球事件应该给中美两国敲响警钟。
Without the safeguard of a hotline mechanism, AI-driven drones and unmanned vessels are highly likely to trigger more accidental escalations in the future.
如果没有热线机制的保障，人工智能驱动的无人机和无人船未来极有可能引发更多意外升级。
Meanwhile, geopolitics and rapid technological iteration—both of which neither side can afford to ignore—are intensifying the security dilemma.
与此同时，双方都无法忽视的地缘政治和快速技术迭代正在加剧安全困境。
Even if China and the United States were to establish a cooperative mechanism on “human control,” they would still face at least three major challenges in practice.
即使中美建立“人控”合作机制，在实践中仍将面临至少三大挑战。
The first is the speed dilemma.
首先是速度困境。
With the help of AI agents, cyberattacks are unfolding at an unprecedented pace.
在人工智能代理的帮助下，网络攻击正在以前所未有的速度展开。
AI-powered defense systems could automatically trigger countermeasures.
人工智能驱动的防御系统可以自动触发对策。
Imagine a scenario where an AI detects abnormal traffic targeting the NC3 system, prompting the defensive AI to automatically disconnect the network or launch countermeasures.
想象一下这样一个场景：AI 检测到针对 NC3 系统的异常流量，促使防御 AI 自动断开网络或启动反制措施。
As a result, the other side might mistakenly interpret this as a cyber intrusion, ultimately pushing both sides’ AI systems into a mode of “automatic confrontation.” AI-enabled cyberattacks can be carried out within milliseconds.
如此一来，对方可能会误将其解读为网络入侵，最终将双方的人工智能系统推向“自动对抗”模式。基于人工智能的网络攻击可以在几毫秒内完成。
In such ultra-high-speed confrontations, the “final human decision” is likely to exceed physiological limits.
在这种超高速的对抗中，“人类的最终决定”很可能会超出生理极限。
Second, there’s the challenge of attribution.
其次，归因方面的挑战。
It is extremely difficult to trace the origin of cyberattacks.
追踪网络攻击的起源极其困难。
Critical infrastructure can be targeted by a state actor or by nonstate actors such as hacker groups, terrorists, or even AI agents themselves.
关键基础设施可能会成为国家行为者或非国家行为者（例如黑客组织、恐怖分子，甚至人工智能特工本身）的目标。
If critical infrastructure falls victim to an AI-powered cyberattack, how can we determine whether the attack was authorized by the government or whether a nonstate actor has abused AI—or even deliberately framed a state actor to provoke conflict?
如果关键基础设施成为人工智能网络攻击的受害者，我们如何确定攻击是否得到政府授权，或者非国家行为者是否滥用人工智能，甚至故意陷害国家行为者以挑起冲突？
Without a reliable attribution mechanism, human accountability cannot be effectively established, and retaliatory actions are all too likely to trigger unintended escalations.
如果没有可靠的归因机制，就无法有效建立人的责任，报复行动很可能引发意外升级。
Third is the security dilemma.
三是安全困境。
Compared to traditional military technologies, the opacity of AI technologies makes it impossible to accurately assess the actual level of development of an adversary’s AI capabilities, the extent to which these technologies can enhance military operational effectiveness, or even to distinguish whether a particular military AI technology is offensive or defensive in nature.
与传统军事技术相比，人工智能技术的不透明性使得无法准确评估对手人工智能能力的实际发展水平、这些技术能够在多大程度上增强军事作战效能，甚至无法区分特定军事人工智能技术是进攻性的还是防御性的。
As a systemic undertaking, the effectiveness of AI’s military applications is further influenced by factors such as the software and hardware integration with various subsystems, human-machine collaboration, military institutional mechanisms, and strategic culture—factors that require a relatively lengthy and complex evaluation process.
作为一项系统工程，人工智能军事应用的有效性还受到各子系统的软硬件集成、人机协作、军事体制机制、战略文化等因素的影响，这些因素需要相对漫长和复杂的评估过程。
As a result, both sides often imagine their adversaries’ AI deployments based on the worst-case scenario and adopt countermeasures that are equally confrontational.
因此，双方经常根据最坏的情况来想象对手的人工智能部署，并采取同样对抗性的对策。
Consequently, threats may be seriously exaggerated, and the spiral of hostility continues to escalate.
因此，威胁可能被严重夸大，敌意不断升级。
These challenges will be difficult to eliminate completely in the near term and can only be managed through continuous strengthening of dialogue and the establishment of safeguards.
这些挑战短期内难以完全消除，只能通过不断加强对话和建立保障措施来应对。
However, cyberattacks represent merely the tip of the iceberg when it comes to military AI risks between China and the United States; many other critical issues also deserve to be incorporated into bilateral military dialogues.
然而，网络攻击只是中美军事人工智能风险的冰山一角。许多其他关键问题也值得纳入双边军事对话。
Among these, lethal autonomous weapons systems (LAWS) urgently require effective control.
其中，致命性自主武器系统（LAWS）迫切需要有效控制。
In the Russia-Ukraine theater and the Middle East, the widespread use of autonomous weapons systems has already led to severe civilian casualties and even heightened the risk of nuclear escalation.
在俄乌战区和中东地区，自主武器系统的广泛使用已经导致严重的平民伤亡，甚至加剧了核升级的风险。
Once LAWS fall into the hands of nonstate actors, the security risks they pose are immense.
一旦法律落入非国家行为者手中，它们带来的安全风险是巨大的。
Given the difficulty in achieving progress on LAWS negotiations within the framework of the United Nations Convention on Certain Conventional Weapons , China and the United States should seek a minimum common ground on “prohibiting fully autonomous lethal attacks” and “ensuring human oversight,” and engage in dialogue and cooperation around measures to prevent the proliferation of LAWS and their misuse by nonstate actors.
鉴于《联合国特定常规武器公约》框架内的LAWS谈判难以取得进展，中美两国应在“禁止完全自主的致命攻击”和“确保人类监督”方面寻求最低限度的共同点，并围绕防止LAWS扩散及其被非国家行为者滥用的措施开展对话与合作。
Throughout the history of U.S.-China relations, there have been numerous accidental incidents involving unmanned vehicles or equipment.
纵观中美关系历史，涉及无人驾驶车辆或设备的意外事件不胜枚举。
Most of these systems are deployed in hotspot regions, and once combined with AI and offensive military capabilities, they could very likely trigger unexpected escalations of crises.
这些系统大多部署在热点地区，一旦与人工智能和进攻性军事能力结合，很可能引发意想不到的危机升级。
Both sides should also promptly explore crisis management and response plans for autonomous weapon systems, working together to uphold regional and global strategic stability.
双方还应抓紧探讨自主武器系统危机管控和应对方案，共同维护地区和全球战略稳定。
The governance of military AI is not a negotiation that can afford to wait—it’s a race against time.
军事人工智能的治理不是一场可以等待的谈判，而是一场与时间的赛跑。
As machines increasingly seize “the right to fire” and even “the right to engage in combat,” China and the United States, as two AI superpowers, bear the historic responsibility of preventing technology from spiraling out of control and safeguarding human civilization.
随着机器越来越多地夺取“开火权”甚至“战斗权”，中美作为两个人工智能超级大国，肩负着防止科技失控、维护人类文明的历史责任。
We look forward to seeing the leaders of China and the United States build on the positive momentum of their ongoing AI dialogue during their September meeting, setting an example for global governance of military AI.
我们期待看到中美两国领导人在9月份的会晤中延续正在进行的人工智能对话的积极势头，为全球军事人工智能治理树立榜样。