We are participating in ForecastBench's October 25 round. We have a few concerns about how scores are calculated for its leaderboard, so we plan to run our own analysis and publish a separate leaderboard. In the spirit of good research practice, we are pre-registering our methods before making our forecasts.
Brier scores. We will score every participant with the standard Brier score.
Correlated questions. We will give correlated questions lower weights, following the approach used in the Metaculus AI Benchmarking tournament.
Early closers. We will only score questions that have reached their scheduled resolution date. Otherwise, questions that can resolve early create an incentive to forecast higher than you actually believe.
Confidence intervals. We will use bootstrapping.
What happens next. We will forecast all 500 questions this weekend. Because we haven't seen the questions yet, we can't assign weights to correlated questions. We plan to do that next week, before any question resolves.
Our goal is a transparent, fair methodology for comparison. We expect the confidence intervals on the Brier scores of the top human forecasters and the top bots to overlap, so no statistically significant conclusions will be possible. Still, we welcome the competition, and we thank the Forecasting Research Institute for running this benchmark.