In September 2026, AI labs pledged to let outside safety testers work inside their walls. Here's what was promised, what evaluators demand, and what's still missing.
Who committed to what
| Who | Commitment | Date |
|---|---|---|
| Anthropic | An embedded external review team "equipped with desks in its offices, access badges, and company laptops" | Sept 12 |
| OpenAI | Sam Altman: "we will do the same" | Sept 12 |
| Anthropic + Accenture | "each expect to invest at least $1 billion over the next five years in independent evaluation of frontier AI" | Sept 18 |
| Meta, Google DeepMind | Have not committed to embedding third-party evaluators (per TechCrunch) | — |
Embedded evaluators are meant to verify safety practices, report incidents, and help assess the alignment of training pipelines. Anthropic proposed they can publish findings "without editorial control by Anthropic."
The five minimum conditions
From the AI Evaluator Forum's public letter (Sept 18, 2026):
- Meaningful independence and full editorial control, with limits on company ownership, commercial ties and contingent pay.
- Differing viewpoints: several evaluation organizations across priority risk areas.
- Transparency about methods, findings, access and terms, with limited NDAs and direct communication with company boards.
- Protection from retaliation, including against retaliatory lawsuits, plus stable funding.
- Access equivalent to the company's own highly privileged employees: systems, data, tools and one-on-one conversations.
AEF-1: the checklist behind it
AEF-1 is the Forum's standard for "Minimum Operating Conditions for Independent Third Party AI Evaluations." Evaluators "complete the checklist and publish it alongside their results." It covers five principles:
- Sufficient access and resources
- Minimized conflicts of interest
- Analytic autonomy
- Transparent methods and results
- Protection of sensitive information
Questions to ask about "tested" claims
- Which evaluator, and who pays them?
- Did they publish an AEF-1 checklist with the results?
- Which conditions were not met, and why?
- Could they publish without the lab's edits?
TechCrunch notes that neither Anthropic nor OpenAI has yet said which evaluators they'll work with, when, or how many. Watch for those details.