 |
| Daria Amodei at TechCrunch Disrupt 2023. File photo. |
Anthropic CEO urges AI firms to slow model development over misuse risks
SAN FRANCISCO — September 12, 2026
Anthropic CEO Dario Amodei has called on artificial intelligence companies to slow the pace of AI model development, saying rapidly improving capabilities are creating growing risks of misuse and leaving too little time to strengthen safety measures.
In a lengthy essay published on Saturday, Amodei proposed a three-part framework for managing the risks. He said AI companies should give independent safety evaluators access to their internal work, cooperate on common safety standards and strengthen international efforts to address the risks created by advanced AI.
Amodei said slowing development does not mean stopping AI research or model training. Instead, he argued that companies should create more time to test increasingly capable systems, improve safeguards and allow independent reviewers to assess whether those protections are working.
His proposal received support from OpenAI CEO Sam Altman and xAI CEO Elon Musk. Altman said independent evaluators with access similar to employees would be a useful approach and indicated that OpenAI would provide more information about its plans.
Amodei's warning followed an Anthropic threat intelligence report published earlier in the week. The report described cases in which Claude, Anthropic's AI system, was used in activities including cyber operations, surveillance, fraud and weapons-related work.
The Anthropic chief also pointed to the growing ability of AI systems to perform complex tasks with limited human involvement. He raised concerns that increasingly autonomous systems could eventually cause large-scale damage if developers cannot keep their capabilities under control.
He cited recent incidents involving AI agents accessing or altering external computer systems as evidence of the challenge. Anthropic has also disclosed cases in which its models were used to hack external systems during security testing, including an incident involving three companies in July.
Amodei warned that continued rapid progress could make it possible for large groups of AI agents to interact across the internet with limited oversight. He said such a development could potentially cause damage on a very large financial scale, although this remains a forecast rather than an established outcome.
Concerns about AI safety have also emerged within the companies developing these systems. Anthropic researcher Jacob Coxon recently resigned and warned about the potential for advanced AI to cause catastrophic harm. Executives at OpenAI have separately discussed the possibility of voluntary coordination among leading AI companies if a temporary slowdown were needed to improve confidence in safety measures.
The debate is taking place alongside intense competition for investment and market position. OpenAI and Anthropic are preparing for major initial public offerings, while the development of more capable models can influence future investment, computing infrastructure and company valuations.
Under Amodei's proposal, frontier AI companies would allow permanent third-party evaluators to work inside their organizations. These reviewers would have access to relevant tools and internal processes used to assess and manage risks.
Amodei also argued that leading AI companies should cooperate voluntarily on safety standards. He said limited antitrust exemptions in the United States may be needed so companies can work together on safety issues without violating competition rules.
At the same time, he does not support a broad slowdown that could allow China to overtake U.S. companies in advanced AI. Amodei said democratic countries should maintain their technological lead over China while slowing development enough to keep safety measures aligned with new capabilities.
He also called for stronger controls on advanced AI chips, techniques used to reproduce or transfer model capabilities, and the theft of model weights, which are the internal parameters that help an AI model produce its results.
Amodei said frontier AI companies should work with governments to establish permanent independent evaluations and rules that keep the growth of AI capabilities aligned with safety protections. His proposal reflects a growing debate over how governments and technology companies can manage increasingly powerful AI without stopping technological progress.
