The number that is not really verifiable
In August 2026, a small team from the safety research group METR walked into Anthropic with a mandate: figure out whether the company's AI was beginning to accelerate its own research. They worked there for eleven weeks. When they published their report in early November, the biggest single finding was not the headline number at all. It was a sentence near the top, quietly admitting that Anthropic had redacted most of the evidence behind that number.
The number they could share is striking. METR concluded Anthropic's models were improving at roughly twice the rate of a year ago, which is about two years' worth of progress compressed into twelve months. The acceleration was not explained by spending more on training compute, and METR attributed the majority of it to AI doing the research. On internal Slack polls, staff who used the lab's most capable models self-reported being around twelve times more productive.
But the number only carries the weight it appears to have because of what METR was not permitted to reproduce. Per METR's own redaction summary, Anthropic redacted "a majority of the evidence we found most informative for these conclusions." The lab justified the cuts by arguing that releasing the detail would help rivals accelerate. In a post on X, co-founder Sam McCandlish wrote that he accepted this made it harder for others to verify METR's work, but insisted the information was commercially sensitive and could hand rivals a roadmap.
The result is a report you can read the headline of and still know almost nothing about the underlying claim. METR put it plainly: because so much of the supporting evidence was redacted, "the public has to rely on METR's judgment much more than would be ideal."
Why this is a vendor risk problem, not a research problem
Most of the chatter about this report has been about AI safety and recursive self-improvement. Those are real concerns for the people who build the models, and they are less useful to the operator licensing a frontier model to run production work. You are not trying to verify METR's read on the pace of AI progress. You are trying to answer a much narrower and more practical question before you sign the licensing deal: how well does this vendor manage the risks that matter to you, and how much of that can you actually check?
Here is the problem. The exact thing METR was unable to publish is the exact thing a buyer most wants to inspect. The redacted material was not a peripheral detail, it was the headline speed of progress and the confidential evidence behind it. METR put it plainly, saying "we also saw redacted evidence that significantly informed our view on this question, and made us take this apparent ≈2x acceleration more seriously." What got cut was the part of the review that should shape your vendor risk assessment.
So you are being asked to trust a vendor's promise about a frontier system while the strongest evidence behind that system has been redacted for commercial reasons. You can ask the vendor to show you its safety audit. They can point you to METR's report and say the outside verdict was positive. But you cannot see the evidence behind that verdict, and the vendor has in this very case demonstrated that it will withhold the evidence even when independence has already done the verification for you.
What a buyer can actually do
You cannot reconstruct METR's redacted analysis, and you will rarely get a frontier vendor to open its books the way Anthropic opened them here. Anthropic shared more information than METR had initially scoped, which is worth noting. Even so, access was restricted, some data requests were denied, and the lab retained the final redaction key. Independent verification is a gift in these engagements, not a right. That limits what you can do, but not nothing.
You can anchor your due diligence on what the vendor can actually produce. Ask a frontier provider, in writing, for the safety assessment behind the capability tier you are being sold. Request the incident history, the red-team results, and the incident record for the specific model class. Confirm, before signing, exactly which parts of any third-party review you are entitled to see, and whether the vendor can grant you the underlying evidence rather than just the summary. A provider that points you to a headline and refuses to let you read the footnotes is giving you the same answer it gave the public.
You can also stop treating a single report as the end of the line. METR's own assessment of bottlenecks shifted "several times" as the team received more detail, and the group was explicit that it was not confident in its final reading. Independent reviews of frontier systems are snapshots, and this one was partly shadowed. Vendor risk is a recurring judgment, not a one-time checkbox, so you should expect annual reassessments, contract clauses that require the vendor to disclose material incidents, and a fallback plan for when the model you licensed underperforms on the safety guarantees you were sold.
You can hedge the concentration risk. Relying on one frontier vendor for your most important work leaves you dependent on a promise you cannot verify. Diversifying across providers, keeping your data and workflows exportable, and standing up your own guardrails around what the model is allowed to do are the controls that survive even when the vendor's report goes red.
How much you can actually verify
The honest answer is that you can verify roughly as much as the public could, which is not much. The independent verdict on Anthropic came with the evidence redacted, and the company treated that redaction as the correct outcome. For an operator, the point is not that verification is impossible. It is that the deepest part of the review is the part you will never see, so the smart move is to build a vendor risk program that stops depending on evidence you are not going to be handed, and treats every frontier promise as something to confirm, not to believe.
Sources
- Why a new report on Anthropic automating AI development is alarming researchers (The Verge, 2026-11-07)
- Independent Assessment of AI R&D Acceleration at Anthropic in October 2026 (METR, 2026-11-05)