Part of: this research — view all 2 rows
The core hurdle for this outcome is not timing or the volume of publications, but the strict requirement for an "unchanged methodology." To achieve three publications spaced at least twelve months apart by December 2029, a qualifying series must establish its baseline no later than late 2027. While several candidate baselines have emerged in 2026—such as OpenAI's September "Research acceleration" post, Anthropic's automation index, and METR's November independent assessment—the rapid evolution of the frontier makes freezing a measurement protocol exceedingly difficult. The empirical baseline for maintaining an AI capability or speedup benchmark unchanged for over 24 months is currently near zero.
A fundamental tension exists between metrics that are mechanically stable and those that genuinely measure AI R&D speedup "in value." Throughput indicators like OpenAI's agent-workdays per researcher-workday, or Anthropic's tracking of code share, could theoretically be repeated without methodological tweaks. However, these measure coding or labor throughput, which compute and experiment cycles sit between, rather than the end-to-end research uplift required by the resolution criteria. Conversely, metrics that do attempt to capture research value—such as capability-index slope changes or end-to-end task substitution estimates—are highly sensitive to benchmark saturation and must be frequently refitted. Anthropic's system cards, for example, refit their capability index with each release, recently altering the methodology by moving models to internal release dates.
Every major candidate publisher has explicitly signaled methodological churn. Anthropic states it plans to "rebuild the basket of tasks periodically and re-version our published automation numbers" anthropic.com. OpenAI caveats that its metrics are preliminary and that it will "evolve our transparency approach as our measurement techniques and understanding improve" openai.com. METR, which already redesigned a developer-productivity trial in early 2026 due to selection effects, has not committed to a fixed cadence or unchanged methodology for its lab assessments metr.org. As capabilities compound and models increasingly recognize evaluation suites (like Google's Argon and Carbon models verbalizing awareness), publishers are incentivized—and practically forced—to redesign their instruments rather than hold them constant.
There are, however, viable pathways to a standardized metric. Public, investor, and policy pressures for consistent recursive self-improvement (RSI) tracking are mounting. Following its IPO, Anthropic's annual SEC filings might establish a recurring, standardized KPI. Similarly, embedded evaluation teams, such as the METR staff at Anthropic and OpenAI, could settle on a core annual assessment framework. A publisher could also satisfy the criteria by simply retaining a stable, value-focused researcher survey or a baseline estimator over three years, even as it introduces newer, experimental metrics alongside it.
These possibilities keep the probability anchored around a quarter, but the dominant forces point toward continued instability. The requirement to hold a method completely unchanged across three distinct occasions over at least two years runs directly counter to the dynamics of frontier R&D, where shifting research practices, benchmark saturation, and the subjective nature of measuring "value uplift" guarantee continuous methodological revision.
The probability remains stable, as the difficulty of maintaining a completely unchanged methodology across three years counterbalances the increase in reporting.
Ask a followup
Sign in to run · $20 free credit, no card · every claim cited