A forecast is only graded when the future arrives. Human forecasting tournaments have lived with this forever: the Good Judgment Project resolved about 500 questions in four years, Good Judgment's professional Superforecasters accumulated 554 resolved questions in eight, and a typical public contest resolves a few dozen a year. (Metaculus has scored about twelve thousand questions over its decade of existence, and notably, much of that volume arrived recently through its own automated benchmark pipeline rather than through the traditional tournament format.)
Each batch of FutureSearch's BTF forecasting question sets holds at least a thousand questions, with 1,900 currently in BTF-3, covering May and June. The questions are already resolved: agents forecast from inside a reconstructed past, researching against a frozen copy of the web, so a full run is graded in hours. That is one to two orders of magnitude more graded forecasts per unit time than a tournament produces.
And yet, even at that volume, the top of our leaderboard clusters. Every competitor on it is a research agent, a frontier model driven through our ReAct harness or its vendor's agent SDK, and every forecast is an agent run of several minutes that reads dozens of pages and costs about a dollar. On a matched set of 1,430 binary questions answered by all eight base agents we track, only 15 of the 28 pairwise accuracy comparisons are statistically significant under a paired bootstrap. The top four agents (two Claude Opus 4.8 configurations, Claude Fable 5, and an agent-harness variant) are mutually indistinguishable: their Brier scores sit within 0.004 of each other, and 1,430 paired resolutions cannot separate them.
The questions needed to detect a difference grow roughly with the inverse square of its size, and each generation of models leaves a smaller difference to detect. Detecting the next improvement costs more than detecting the last one, indefinitely.
Which makes forecaster development expensive
If you want to grade a change to a forecasting system, you have two options. You can wait for the future, which is slow, and nothing stays constant while you wait. Or you can pastcast at scale, which is what BTF-3 is for, and which took us a benchmark of frozen web corpora with tens of thousands of pages per question and an automated question-generation and resolution pipeline to build. Either way, statistical resolution is bought with infrastructure, money, and time.
Horizons make it worse, and not only because of the waiting. Our questions resolve within weeks or months, which is what lets pastcasting work: the questions are recent enough that their outcomes postdate the models' training data. Now try the same trick on a ten-year question. For the outcome to be known today, the forecast date has to sit ten years in the past, and every current model carries those ten years in its weights, so it would be remembering the decade rather than forecasting it. Grading a frontier model's decade-scale accuracy means waiting the decade.
Checking the worldview instead of the outcomes
There is information in a set of forecasts that does not require knowing any outcomes: whether the forecasts hang together.
The crude version of this is logical. P(X) and P(not X) ought to sum to one, or nearly so. If A implies B, then P(A) must not exceed P(B). Checks like these have been studied as a way to evaluate forecasters without resolutions, and they do catch bad forecasters, but only very bad ones. Frontier agents rarely fail them outright.
The interesting inconsistencies are not logical errors. Suppose an agent says P(Russia invades a European country other than Ukraine) = 8%, and also says P(Russia invades Austria) = 7%. No law of probability is violated: Austria is one such country, and 7 is less than 8. But taken together the two numbers claim that conditional on Russia invading someone, it is almost certainly Austria, which is absurd, and noticing that requires no forecasting talent at all. A sensible repair nudges the first number up a little and the second down a lot.
Our world-modelling step does this at scale: it identifies inconsistencies in the latent worldview behind an agent's forecasts, the common-sense extension of the logical checks above, and repairs them. Since the step records every forecast before and after, it yields a natural measure of how consistent an agent's worldview was in the first place: the total probability mass the repair had to move. We express it in probability points moved per question. Lower means the agent's forecasts already told one story.
A forecast set can only be consistent-but-wrong if it tells a coherent story about a world that is not ours. Producing one of those takes effort: every individual forecast must be defensible on its own, and they must all be wrong together, in a coordinated direction. If an agent's individual forecasts are each reasonable (frontier agents clear this bar; an agent that answered 50% on everything would be perfectly consistent and useless, which is why the measure leans on the forecasts not being individually horrible) then coherence is most of what is left to get right. If that is true, more consistent agents should be better forecasters, and they are.
The most consistent agent on the benchmark, built on Claude Opus 4.8 (xhigh), needed only 0.68 points moved per question and is also the most accurate. The least consistent, built on Claude Sonnet 5 (xhigh), needed 1.30, nearly twice as much, and is also the least accurate. In between, the rank correlation is 0.67, and asking for an exact ranking is the wrong test anyway: accuracy cannot produce one either. The right comparison is between the clusters each measure can actually resolve, and those match. On our leaderboard, four agents are statistically indistinguishable from first place; the consistency measure's top cluster contains three of those four. At the other end the agreement is exact: both measures put the Sonnet 5 agent alone at the bottom, and consistency says so more firmly than accuracy does (on this question set, Brier scores cannot separate the bottom two agents; consistency separates Sonnet from all seven others). Overall the consistency measure statistically separates 16 of the 28 agent pairs, one more than Brier scores separate, and it does so having never seen a resolution.
The measure needs a set of related questions and an agent's forecasts on them. It does not need the future to arrive. Nothing about it privileges 30-day questions over 30-year ones. We would not put decade-horizon trust in any single number, but as a leading indicator of which agents to trust on horizons that cannot be graded within a development cycle, we think consistency is the best signal available today, and we are adding it as a standing component of our evals.
When the two rankings disagree
What about an agent that scores a decent Brier but sits low on the consistency ranking? That disagreement is good news, because the repair step doubles as a lever: fixing the inconsistencies in a reasonable manner should improve the forecasts, and it costs compute, not months of waiting.
We scored every agent's repaired forecasts against the same resolutions as the originals:
The pass improved all eight agents. The gains are small in absolute terms, between 0.0008 and 0.0021 Brier, but so are the gaps at the top of the leaderboard, and the two largest gains (GPT-5.6 Sol and GPT-5.5, both around 0.002) are individually significant at p around 0.001 on the paired bootstrap. For scale: 0.002 of Brier is roughly the gap between fourth and sixth place on this table.
We cannot yet tell you the exchange rate between inconsistency and recoverable accuracy. The direction is established (the repair helped all eight agents, and the most consistent agent gained least, exactly as a headroom story predicts), but the per-agent gains are so close together that none of the 28 pairwise differences between them reaches significance on 1,430 questions; estimating how the gain scales with inconsistency needs more data than establishing that there is one. Two details do point the way the headroom story predicts. The measure's premise, remember, is that the individual forecasts are not horrible, and the one agent that strains that premise, Sonnet 5, is also where the repair works hardest for the least return: it extracted less accuracy per point of probability moved there than on any other agent (a repair built from weak neighbouring forecasts has less to anchor on). Set Sonnet aside and the correlation between inconsistency and gain turns positive, though with seven agents it is nowhere near significant. We will report the exchange rate when we can measure it.
The practical upshot for anyone building on top of these agents: the consistency ranking tells you which agents have headroom that post-processing can recover, and the repaired forecasts are simply better than the raw ones. An agent that is accurate but inconsistent is one you can improve today. An agent that is accurate because it is consistent has already priced its worldview in, and beating it will require actual new information about the world.
That, of course, is what the top of every leaderboard eventually looks like.
The pass behind this analysis is the world-modelling step described in The shared world model behind every forecast; the evaluation approach is described in Iterating on a forecaster without waiting for the future. Live standings and benchmarks are at evals.futuresearch.ai.