System.Xml.XmlElement
One of the key themes at the AI Engineer New York conference this week was how to balance agent autonomy with security and safety .
本周人工智能工程师纽约会议的关键主题之一是如何平衡代理自主性与安全性。
Both frontier labs and financial firms focused on this issue in their presentations this week.
前沿实验室和金融公司在本周的演讲中都重点关注了这个问题。
The main lesson: safe autonomy depends on where you put controls and human judgment , rather than how many restrictions you impose.
主要教训是：安全自主取决于控制措施和人为判断的位置，而不是施加多少限制。
Adding commerce to an agent usually means rebuilding search and checkout from scratch.
向代理添加商务通常意味着从头开始重建搜索和结帐。
shop.dev gives you a single API for shopping in any agent or app.
shop.dev 为您提供了一个 API，可以在任何代理或应用程序中进行购物。
One connection gives you access to over a billion products with current price and availability, instant sign in with Shop, and one-tap Shop Pay checkout that happens right in the conversation.
通过一次连接，您可以访问超过 10 亿种具有当前价格和库存情况的产品，即时登录 Shop，并在对话中一键进行 Shop Pay 结账。
It’s open to any developer, with no account or key needed to try it.
它对任何开发者开放，无需帐户或密钥即可尝试。
→ Try shop.dev today
→ 立即尝试 shop.dev
Joseph Kim , Head of ChatGPT for Financial Services at OpenAI, gave a keynote at AIE New York today.
OpenAI 金融服务 ChatGPT 主管 Joseph Kim 今天在纽约 AIE 上发表了主题演讲。
(We also did a special podcast with Kim, which will be released shortly.) He talked about reliability and a “threshold of usefulness,” which he said is “the point where a model becomes capable and reliable enough to save you time.”
（我们还与 Kim 一起制作了一个特别的播客，该播客将很快发布。）他谈到了可靠性和“有用的阈值”，他说这是“模型变得足够强大和可靠以节省时间的点”。
Kim argued that ever-growing model capability will soon take financial analysts over the threshold and allow them to delegate more work to agents.
金认为，不断增长的模型能力很快就会使金融分析师跨过门槛，并允许他们将更多工作委托给代理人。
He made some specific predictions about AI’s role in finance in 2027.
他对2027年人工智能在金融领域的作用做出了一些具体预测。
Not only will the financial industry rely less on Excel and PowerPoint next year, he claimed, but agents will be good enough to do “investment idea generation and trading.”
他声称，明年金融业不仅会减少对 Excel 和 PowerPoint 的依赖，而且代理商也将有足够的能力进行“投资想法生成和交易”。
The “rising tide” of model capabilities is also leading to better traceability .
模型能力的“崛起”也带来了更好的可追溯性。
“In finance, you need to be able to check where a number came from and understand the context around it,” said Kim.
“在金融领域，你需要能够检查数字的来源并了解其周围的背景，”金说。
“We wanted that review to be a natural part of the [ChatGPT for Financial Services] experience.
“我们希望该审核成为 [ChatGPT for Financial Services] 体验的自然组成部分。
These are the kind of product details that help turn a model’s capabilities into work that someone can use.
这些产品细节有助于将模型的功能转化为人们可以使用的工作。
”
”
As for when a human needs to be in the loop, Kim framed this as a product design decision.
至于何时需要人类参与其中，Kim 将其视为产品设计决策。
“How would the interface change?
“界面会如何变化？
What would the user still need to control, review, or approve?
用户还需要控制、审查或批准什么？
Those are product design questions you can start working on now.”
这些是你现在就可以开始解决的产品设计问题。”
John Feminella , SVP Architecture at the hedge fund Two Sigma , is tackling the exact kinds of product design questions Kim referenced.
对冲基金 Two Sigma 的架构高级副总裁 John Feminella 正在解决 Kim 提到的确切类型的产品设计问题。
In a keynote, he talked about five of the controls they’ve implemented to keep agents in check without limiting model capabilities.
在主题演讲中，他谈到了他们为在不限制模型功能的情况下控制代理而实施的五种控制措施。
Feminella argued that safety and innovation shouldn’t be viewed as opposite ends of a spectrum.
费米内拉认为，安全和创新不应被视为两个极端。
He was especially critical of “approval fatigue,” where a person approving everything can hold back innovation — and also introduce risk, if they’re not fully paying attention to all the decisions they’re required to make.
他特别批评“批准疲劳”，即批准一切的人会阻碍创新，如果他们没有充分关注需要做出的所有决策，还会带来风险。
At Two Sigma, they’ve implemented a set of controls , which he defined as “guardrails and shared capabilities that let teams move in ways that don’t hinder what they’re trying to do.”
在两西格码，他们实施了一套控制措施，他将其定义为“护栏和共享功能，让团队以不妨碍他们尝试做的事情的方式行动。”
The controls are composable, meaning “each one does one job and then together they let us run agents with real autonomy at an appropriate level of safety.”
这些控件是可组合的，这意味着“每个控件只做一项工作，然后它们一起让我们在适当的安全级别上运行具有真正自主权的代理。”
One of the controls is to use auto-mode classifiers , which Feminella said “make decisions about routine calls and then send a person only the requests that are most important, the ones that are most likely to make a difference.”
其中一种控制方法是使用自动模式分类器，费米内拉表示，“对日常呼叫做出决定，然后只向人员发送最重要、最有可能产生影响的请求。”
Classifiers can be used for ambiguous cases, whereas deterministic rules are better for clear cases.
分类器可用于不明确的情况，而确定性规则更适合明确的情况。
Anthropic’s presentation outlined a similar framework, in its case with the goal of building “trust” into financial agentic workflows.
Anthropic 的演讲概述了一个类似的框架，其目标是在金融代理工作流程中建立“信任”。
Last month, Anthropic announced Claude for Financial Advisors , allowing financial advisors to connect their systems to Claude and a set of workflow skills.
上个月，Anthropic 宣布推出 Claude for Financial Advisors，允许财务顾问将他们的系统连接到 Claude 和一组工作流程技能。
At AIE this week, Belinda Neal , Managing Director of Financial Services at Anthropic, explained why Anthropic has partnered with financial firms on the new product.
在本周的 AIE 上，Anthropic 金融服务董事总经理 Belinda Neal 解释了 Anthropic 与金融公司合作开发新产品的原因。
“Every so often, a model crosses a threshold.
“模型时常会跨越门槛。
And when it does, a wave of working changes.”
当它发生时，工作方式就会发生一波变化。”
Neal’s colleague Keon Shahab , who works on applied AI at Anthropic, then talked about “building the trust layer for the agents.” He said this involves three steps:
Neal 的同事 Keon Shahab 在 Anthropic 从事应用人工智能工作，他随后谈到了“为代理建立信任层”。他说，这涉及三个步骤：
“Who the agent is acting as with an identity and permissions layer .
“代理在身份和权限层中充当谁的角色。
How we can sandbox and determine the approval layer of the agent, so that someone is signing off before it acts.
我们如何沙箱化并确定代理的审批层，以便有人在其行动之前签字。
And lastly, how we know the agent will still work once we adjust any variables over time.”
最后，我们如何知道一旦我们随着时间的推移调整任何变量，代理仍然会工作。”
Shahab illustrated the boundary between proposing and acting with a trade break investigation agent, which would recommend an order amendment for human approval.
沙哈布说明了提议和与贸易中断调查机构合作之间的界限，该调查机构将建议一项命令修改以供人工批准。
“We want it [the agent] to just propose new amendments to our orders data, not actually make those amendments,” he said.
“我们希望它（代理商）只是对我们的订单数据提出新的修改，而不是实际进行这些修改，”他说。
“By building once with a foundational layer that you trust , you can work at speed to deploy many agents at scale,” Shahab concluded.
“通过使用您信任的基础层进行一次构建，您可以快速大规模部署许多代理，”Shahab 总结道。
As Head of Architecture & Emerging Technologies in the Office of the CTO at Bloomberg, Andrey Rybka has the responsibility to balance autonomy and scale at the company.
作为彭博社首席技术官办公室架构和新兴技术主管，Andrey Rybka 负责平衡公司的自主权和规模。
In a presentation today at AIE, Rybka talked about a series of real-life events where agents caused havoc — and the defenses that can help prevent it .
今天在 AIE 的演讲中，Rybka 谈到了特工造成严重破坏的一系列现实事件，以及可以帮助防止这种情况的防御措施。
These include scoped identities and credentials, controls over tool calls, outbound traffic restrictions, audit trails, budgets, and kill switches.
其中包括范围内的身份和凭证、对工具调用的控制、出站流量限制、审计跟踪、预算和终止开关。
In one example, an agent treated instructions embedded in a marketplace review as an instruction to stock a product.
在一个示例中，代理将市场评论中嵌入的指令视为库存产品的指令。
Rybka recommended input validation, prompt injection defenses, provenance tracking, and human approval for high-value actions.
Rybka 建议对高价值操作进行输入验证、即时注入防御、来源跟踪和人工批准。
“We want deterministic controls outside the model, ” Rybka said.
“我们希望在模型之外进行确定性控制，”雷布卡说。
“Check every call, log it, and keep a kill switch.”
“检查每一个呼叫，记录下来，并保留终止开关。”
Even though frontier labs like OpenAI and Anthropic are building a ‘trust framework’ for financial agents, ultimately it’s up to financial firms to put controls around their agents .
尽管像 OpenAI 和 Anthropic 这样的前沿实验室正在为金融代理构建“信任框架”，但最终还是要由金融公司对其代理进行控制。
Two Sigma’s John Feminella hit the nail on the head: implement controls that enable autonomy to happen, but don’t put up roadblocks.
两个西格码的约翰·费米内拉一针见血：实施控制措施，实现自动化，但不要设置路障。
Or as he put it in his conclusion: “Why do cars have brakes?
或者正如他在结论中所说：“为什么汽车有刹车？
It’s not just so that you can stop, it’s so that you can go fast and stop safely.”
这不仅是为了让你能够停下来，也是为了让你能够走得快并安全地停下来。”