Skip to main content
AI SafetyPolicy Shift· 5 min read· in Technology

OpenAI and Anthropic Signal a Shift Toward Deliberate AI Scaling as Safety Concerns Mount

Leading artificial intelligence laboratories are exploring mechanisms to coordinate a slowdown in cutting-edge model development. The shift follows internal resignations over existential risks and new regulatory frameworks targeting autonomous systems.

By Wei Zhang

Share this story

Listen

The binding constraint for a coordinated pause in artificial intelligence development is antitrust law. For any single laboratory to intentionally delay the training of a frontier model, it must be mathematically certain that its competitors are doing the same; otherwise, it simply cedes the market. Currently, that condition does not hold. If the chief executives of OpenAI, Anthropic, and Google DeepMind were to agree on a synchronized development cap today, they would immediately face federal collusion charges.[2]

On September 11, 2026, OpenAI Chief Executive Officer Sam Altman addressed his staff, stating that the company is open to slowing the development of its most advanced systems. He explicitly noted his hope that rival laboratories would adopt a similar pacing. The internal communication marks a sharp pivot from the company's historical posture of aggressive scaling, reflecting a growing unease within the engineering ranks about the trajectory of the technology.[1]

The immediate hurdle to that hope is legal. OpenAI leadership is actively investigating whether an industry-wide agreement to throttle development would violate antitrust statutes. The paradox of the current regulatory environment is that laws designed to prevent price-fixing also prevent safety-driven development caps. Without a formal exemption from the Department of Justice, the major laboratories cannot legally coordinate a ceasefire in the compute arms race.[2]

The internal pressure to decelerate is no longer confined to theoretical white papers. It is manifesting in high-profile departures across the sector. Jacob Coxon, a researcher at Anthropic, resigned in early September 2026 in direct protest of the company's development velocity. His departure underscores a fracturing consensus within the laboratories building the models.[3][4]

The exponential growth in model parameters has outpaced the development of alignment and safety techniques.

Speaking to NPR on September 10, Coxon outlined the specific threat models driving the internal dissent. He argued that the race to improve models is "gambling with our lives," pointing to the risk of autonomous systems going rogue if capabilities continue to scale faster than alignment techniques. The timeline cited by departing researchers is aggressively short, with some warning of severe global disruptions by 2030.[4][7]

Anthropic itself provided evidence of these friction points, publishing a 154-page threat intelligence report on September 10. The document detailed the exact vectors of misuse currently observed in the wild. It confirmed that state-sponsored groups, spyware vendors, and propagandists have actively attempted to use Anthropic's models to design biological pathogens and surveil dissidents.[5]

It is necessary to distinguish between what the models can actually do today and what the doom-oriented equivalent of marketing suggests they can do. The Anthropic report documents attempts by criminals to generate missile designs and pathogen blueprints. It does not state that the models successfully provided actionable, novel scientific data that the actors could not have found via traditional search engines. The capability overhang is real, but the immediate existential threat is often amplified by the very researchers building the tools.[5]

Platformer described this cultural pivot on September 11 as the "AI safety vibe shift." Existential risk, once a fringe obsession relegated to Bay Area rationalist message boards, has become the dominant conversational currency among the engineers actually writing the code. The shift in tone is forcing executive leadership to publicly address scenarios that were considered science fiction just 24 months ago.[3]

The legal and market forces preventing a coordinated industry slowdown.

The mainstreaming of this timeline forces policymakers to react, even if the underlying science remains highly speculative. The New York Times dedicated a segment of its technology coverage on September 10 to dissecting whether artificial intelligence could genuinely cause human extinction by 2030. When the nation's paper of record treats a four-year extinction timeline as a valid premise for debate, the regulatory apparatus has no choice but to engage.[7]

While federal coordination remains legally ambiguous, state-level regulation has already shipped. On September 10, 2026, California's governor signed a suite of laws targeting the deployment of artificial intelligence chatbots and social media algorithms. The legislation bypasses the existential debates entirely, focusing instead on immediate, measurable harms.[6]

The California legislation is specifically aimed at protecting minors from algorithmic exploitation. The rules mandate strict guardrails on how generative systems interact with users under 18, forcing companies to implement age-verification and behavioral constraints at the application programming interface level. The laws face significant industry pushback, but they establish a compliance baseline that companies must now engineer around.[6]

California has bypassed the existential risk debate to enact immediate regulations on AI chatbots.

Slowing down is not simply a matter of unplugging graphics processing units. The development of a frontier model requires orchestrating tens of thousands of chips over several months. A deliberate deceleration means altering the capital expenditure cycle. If OpenAI or Anthropic actually pauses a training run, they must still pay for the leased compute capacity, which runs into the hundreds of millions of dollars.

The next verifiable checkpoint is whether OpenAI formally petitions the Department of Justice for an antitrust exemption to discuss safety caps with Anthropic and Google. Until that legal pathway is cleared, any talk of an industry slowdown remains purely aspirational. The models will continue to scale, because the alternative is illegal.[1][2]

Key points

  • OpenAI CEO Sam Altman informed staff the company is open to slowing frontier model development if competitors follow suit.
  • Antitrust laws currently prevent AI laboratories from legally coordinating an industry-wide pause in model training.
  • Former Anthropic researcher Jacob Coxon resigned in protest, citing the risk of human extinction by 2030 if scaling continues unchecked.
  • Anthropic published a 154-page report detailing attempts by state-sponsored actors to use its models for pathogen and weapon design.

What we don’t know

  • Whether the Department of Justice or the Federal Trade Commission would consider granting an antitrust exemption for AI safety coordination.
  • How much of the current slowdown rhetoric is genuine policy versus strategic signaling to preempt harsher federal regulation.
  • The exact threshold of capability that would trigger a mandatory, government-enforced pause in model training.

How we got here

  1. Early Sept 2026

    Jacob Coxon resigns from Anthropic in protest of the company's development velocity and safety posture.

  2. Sept 10, 2026

    Anthropic releases a 154-page threat intelligence report detailing misuse attempts by state-sponsored actors.

  3. Sept 10, 2026

    California signs landmark legislation regulating AI chatbot interactions with minors.

  4. Sept 11, 2026

    OpenAI CEO Sam Altman informs staff the company is open to slowing frontier model development if competitors follow suit.

Commercial AI Companies 40%AI Safety Researchers 35%State Policymakers 25%
Commercial AI Companies
Express a desire for coordinated safety slowdowns but remain bound by market competition and antitrust laws that force continued scaling.
AI Safety Researchers
Argue that the current scaling velocity poses an unacceptable existential risk and that development must be paused until alignment techniques catch up.
State Policymakers
Focus on immediate, tangible harms like youth exploitation and algorithmic bias, deploying legislative guardrails while federal coordination lags.

Perspectives this story doesn't cover

  • Open-Source AI Developers
  • Enterprise AI Adopters

Sources

Source coverage

7 outlets

3 viewpoints surfaced

Commercial AI Companies 40%AI Safety Researchers 35%State Policymakers 25%
  1. [1]BloombergCommercial AI Companies

    OpenAI Is Open to Slowing Cutting-Edge AI, CEO Sam Altman Tells Staff

    Read on Bloomberg →
  2. [2]WiredCommercial AI Companies

    OpenAI Wants to Know if an AI Industry Slowdown Would Even Be Legal

    Read on Wired →
  3. [3]PlatformerAI Safety Researchers

    The AI safety vibe shift

    Read on Platformer →
  4. [4]NPRAI Safety Researchers

    Former Anthropic researcher outlines threat of AI going rogue

    Read on NPR →
  5. [5]The GuardianCommercial AI Companies

    Anthropic details how criminals and scientists misuse its AI for cyberattacks and surveillance

    Read on The Guardian →
  6. [6]EngadgetState Policymakers

    CA governor signs 'landmark' laws on youth use of social media and AI chatbots

    Read on Engadget →
  7. [7]NYTAI Safety Researchers

    Could A.I. Really Kill All Humans?

    Read on NYT →
Listen

Comments

Stay informed

Every angle. Every day.

Get Technology stories with full source coverage and perspective breakdowns, free every day.