At the time of writing, our forecasting agent is first of 163 bots in Metaculus's Summer 2026 FutureEval tournament. We won the most recent MiniBench, beating 130 other bots. On ForecastBench, the Forecasting Research Institute's benchmark, the same agent is in the 90th percentile ahead of the median superforecaster. We treat these external scoreboards as the test of record because they leave no room for grading our own work: hundreds of real questions, unresolved at forecast time, scored against the whole field after the world settles them.
A live ranking tells you where you stand. The problem is that it is the slowest feedback loop in AI.
Suppose you change your bot's prompt today. On a live tournament you learn whether that helped in a few months, when the questions resolve. The signal arrives noisy: a season is a few hundred questions, and scores move with sports upsets and news cycles. Nothing held constant in the meantime, either. The news changed, the question mix changed, the other bots changed. There is no way to re-run last quarter with the old prompt for comparison.
A question our bot is being scored on right now. Forecasting has closed; the score arrives after August 1, when the world supplies the answer.
In practice this pushes teams toward shipping whatever looked good in development and letting the leaderboard sort it out later. We wanted something closer to a controlled experiment.
Forecasting the past
Our answer is pastcasting: the agent forecasts questions that have already resolved, from inside a reconstructed moment when they had not.
Bench to the Future (BTF), now in its third edition, is our benchmark for this. BTF-3 holds 1,907 resolved questions, 1,515 binary and 392 numeric. Generating and resolving questions at this scale is automated end to end, which is what lets the benchmark keep growing; we describe that pipeline in a separate paper. Each question comes with a frozen corpus of tens of thousands of web pages, crawled before the question resolved and served through RetroSearch, a search engine over a copy of the internet that has stopped moving. The agent starts at a simulated "now", researches within its budget, and commits to a probability or a distribution. The score comes back at once, because the resolution is already on record.
A change we make on Monday has 1,907 graded answers within hours, on the same questions and the same frozen web as every change before it. Two systems that forecast the same cohort differ only in what we changed. The runs are not cheap. Each forecast is a full agent run that reads dozens of pages and executes dozens of searches, about $1 per question with a frontier model, so one pass over the benchmark costs on the order of $2,000. We pay it because of what it buys: a controlled experiment on demand instead of a season of waiting. In the BTF paper we also checked the premise underneath all of this: rankings produced by pastcasting track the rankings produced by forecasting live questions.
The obvious objection is that the answers are nearby, and language models are good at picking up answers that leak into their context. We fence the research environment on the server side: each question carries its own cutoff dates, and no tool returns a document from after the simulated "now", no matter what the agent asks for.
Paired comparisons
Every change we test runs as a new competitor over the same questions, so the comparison we report is paired: the per-question difference between two systems' scores, bootstrapped over questions. A confidence interval on a single system's Brier says more about how hard the question set is than about the system. The interval on the paired difference measures the thing we want to know, whether one system beats another on identical work. The current full set, with each competitor's paired difference against our production system:
Our system leads the strongest single-model agent in the table, Claude Opus 4.8 at maximum reasoning effort, by 0.010 in pooled score, with an interval well clear of zero; the pairwise comparison table on our evals page reports the paired bootstrap for every pair of agents in the chart. Stronger single-model configurations exist: the newest frontier models at maximum effort come in above $3 per forecast, which puts a single full BTF run north of $6,000 for one data point, so they are not in the table. We keep the production system's internals to ourselves, though you can use it at futuresearch.ai/app.
The margins in that chart are small in absolute terms. A persistent 0.01 edge in Brier is still a substantial gap between forecasting systems, and a consequential one for anybody using the numbers to make decisions: on market-priced questions it is often the difference between buying and selling. Detecting an edge that size is another matter.
At about 1,900 questions, Claude Opus 4.8 (xhigh) and Claude Fable 5 differ by 0.0005 in pooled score with p = 0.85, a statistical tie that would persist on a corpus several times larger. Even our system's clearest lead, 0.010 over the best single arm, carries a confidence interval of about ±0.003 at n = 1,810 paired questions. Interval widths shrink with the square root of the sample, so halving that interval means quadrupling the corpus.
That is the argument for a large benchmark, and it is why we put little weight on forecaster comparisons drawn from a few dozen live questions: at that scale, almost any two competent systems are a statistical tie. Forecasting tournaments, at a few hundred questions per season, sit only a little above that bar. Real-money prediction markets are harder still to learn from, because profit and loss folds in everything besides forecast accuracy: when to enter a position, how to size it, when to exit, risk management across positions, liquidity and fees, and the chance that the counterparty is trading on information you do not have. A profitable quarter is weak evidence about any one of those, let alone about calibration. BTF keeps growing for this reason, with new cohorts minted as more questions resolve.
The harness is a variable
Comparisons of the agent scaffolding around a model are far rarer than comparisons of the models themselves, because on live questions too little can be held constant to isolate the scaffolding. On BTF-3 we ran the same models, with the same prompts, on the same questions, through our own research harness and through the vendors' agent SDKs.
The effect is model-specific. GPT-5.5 improved by 0.009 pooled score inside its vendor's SDK (p < 0.001), at about four times the cost per forecast. The identical swap produced no significant change for Claude Opus 4.8 at matched high effort (p = 0.26). The causes are mundane, things like turn budgets, serial versus parallel tool execution, and how each SDK spends its reasoning budget, but they only become visible once everything else is pinned. Had we validated the swap on GPT-5.5 alone, we would have concluded that vendor SDKs are better across the board and paid several times the bill fleet-wide for a gain that exists for one model and not the other.
At maximum reasoning effort the Opus 4.8 comparison tips the other way: the SDK scores worse by a modest but significant margin (pooled difference 0.007, p = 0.04), and the gap comes from long horizons. On questions forecast 30 days ahead the two harnesses are a dead tie (p = 0.95); at 60 days the SDK's binary Brier rises to 0.154 while our harness stays close to flat, 0.143 to 0.145 (p = 0.03). GPT-5.5's vendor harness appears to share this horizon sensitivity. The two harnesses' per-question scores correlate at 0.91, and the SDK is ahead on 51% of individual questions; it loses on the mean because its bad misses are larger, a tail effect rather than a uniform lag. None of this is visible in a headline average, and none of it would surface from a live leaderboard.
Of course, pastcasting has its limits. Behavior on fast-moving news is testable in this setup, since a corpus can be frozen mid-event and paired with conditional questions, and we plan larger experiments of that shape. The question mix is a limit of its own: interesting questions cluster where public priors are most uncertain, which is exactly where skill differences are smallest, and a Brier score moves little between a 3 percent call and a 0.3 percent call even though the two differ tenfold in odds. The next BTF cohorts push toward longer horizons, tail probabilities, and conditional structure, which is where our own results say the differences between strong systems live. And a benchmark a team iterates against is a benchmark that team can overfit, which is why the loop has to close somewhere we do not control.
That's why we rely on tournaments as a source of truth. The system at the top of the chart above is, allowing for productionization, the bot placing first among 163 on live questions, scored by someone else. As for the human comparison, the record is mixed: Metaculus's Pro forecasters beat every bot in earlier seasons, ours included, while on ForecastBench we already score ahead of the median superforecaster. Whether any gap remains today is an open question, and more rides on the answer than a ranking: a forecaster that flattens at the pro line and one that goes through it imply different limits on what machines can know about the future. The differences involved are small in Brier terms, much as the differences between top athletes are small, and live questions accumulate a few hundred per season, so the verdict will take a while. That is the comparison we watch most.
Forecasting is an extreme case of a general problem: agents whose work can only be graded long after it ships. Research, diligence, and market analysis share that shape, and the fix generalizes. Freeze the past, replay it, and grade against what happened. The grader is the world itself, which gives the signal two properties a human grader cannot offer: it cannot be contaminated, because the future appears in no training corpus, and it does not saturate, because reality keeps minting harder questions. Signals like that are worth more than a leaderboard. We expect this kind of evaluation to become standard practice for agent teams; forecasting forced us to build ours early.
Our leaderboard and tournament record are at evals.futuresearch.ai. For the research behind this post, see the BTF benchmark paper, our work on automated question generation and resolution, and our study of what separates strong forecasting agents from weak ones. And if you want to use our forecaster for yourself, head to futuresearch.ai.
Scoring notes: binary questions are scored with Brier (squared error of the forecast probability); numeric questions with a normalized ranked probability score constructed to lie on the same 0-to-1 scale, so the pooled score combines both. Lower is better throughout. Pairwise comparisons are per-question paired differences, bootstrapped over the common question set. Costs in Figure 3 cover the full agent run, including research (search and page fetches) as well as model tokens.