Our Correlation Weights for ForecastBench's Oct 25 Round

Last week we pre-registered how we will score ForecastBench's October 25 round, including lower weights for correlated questions. We promised to set those weights before any question resolves. This post publishes them.

Why weights matter. About 50 of the 250 market questions, roughly 20%, will be settled by the US midterm elections on November 3. That would be fine if the election questions were independent, but history shows that results across races are often highly correlated. The polls may turn out to be skewed toward Democrats or toward Republicans, and that polling error moves many races at once. Instead of 50 independent questions, the election questions may be worth the equivalent of 5 to 10. The same concern applies to some dataset questions: sixteen of them track points on one euro-area yield curve. We try to account for this by giving correlated questions lower weights.

Groups. We put each of the 500 questions either in a group or on its own. Questions share a group when one event or one number settles them, such as candidates in the same race or thresholds on the same CPI release. A few larger groups share one dominant driver, like the national swing in the midterms or the euro yield curve. We tried to group only questions we expect to be correlated, but these calls are subjective and some will be wrong. When in doubt, we erred on the side of treating a question as independent. The result is 20 groups covering 151 questions, with the other 349 standing alone. The groups will not change. We don't love this system, but we think it is fairer than treating every question as independent.

Weights. We use the formula from the Metaculus AI Benchmarking tournament. In a group of N questions, each question gets a weight of log₂(N+1)/(N+1). A question on its own gets 1. A pair gets 0.53 each, so together they count as about one question.

Market questions have one forecast each, so a market question's weight is the weight of its single forecast.

Dataset questions have five forecasts each, one per resolution date, and each forecast is scored separately. One question, for example, asks whether a particular euro-area government bond yield will be higher than it was on the due date, checked on November 1, November 24, January 23, April 23 and October 25, 2027. The five forecasts on a question move together, so we treat them as a group of five under the same formula. Each gets 0.431, and together they add up to 2.15. A dataset forecast's weight is therefore its question's weight times 0.431. Four FRED questions have only four resolution dates, so their forecasts get 0.464 instead.

Take the euro yield group. It has 16 questions, each about a different yield on the same curve, and each question has five forecasts, so the group has 16 × 5 = 80 forecasts. With 16 questions in the group, each question's weight is log₂(17)/17 = 0.240. Each of the 80 forecasts gets 0.240 × 0.431 = 0.104, compared with 0.431 for a forecast on a question that stands alone. Together, the 16 questions count as 3.8 questions on their own.

Market and dataset forecasts are scored in separate halves (see Scoring below), so a weight only matters relative to other weights in the same half.

The tables at the end list every grouped question. Any question missing from them stands alone, with weight 1: a single forecast at weight 1 for a market question, or five forecasts at 0.431 for a dataset question.

The human questions. ForecastBench gave humans 200 of the 500 questions: 130 market and 70 dataset. It published the list on October 26 at 20:45 UTC, shortly before this post first went up. We set the groups without seeing it. For the human comparison we apply the same formula to the same groups, counting only the group members that are in the human set. For example, 27 of the 52 midterm questions are in the human set, so each of them gets log₂(28)/28 = 0.172 there. A grouped question whose group has no other member in the human set stands alone there, with weight 1.

Scoring. For each participant we compute a weighted average Brier score over market forecasts and, separately, over dataset forecasts, counting only forecasts that have reached their scheduled resolution date. The final score gives the two halves equal weight.

Our two extra entries. ForecastBench allows each team up to three entries per round. Besides our main entry, futuresearch_v6, we entered futuresearch_v6_dem_lean and futuresearch_v6_gop_lean. Both match our main forecasts except on 51 midterm questions. There, the first doubles the odds of the outcome that favors Democrats and the second doubles the odds of the outcome that favors Republicans. We entered them so we would not start at a disadvantage. Suppose a weaker forecaster in this round is biased toward Democrats and another is biased against them. Whichever bias the election rewards will likely score very well on those questions, and the rest of the field may have no way to catch up. Our weighting is meant to soften that effect, though it is far from perfect. We will leave both lean entries out of our own analysis.

All groups

The tables below list every question that belongs to a group. Any question not listed stands alone and has weight 1 in both columns.

• Overall weight is the question's weight when we score all 500 questions.
• Human weight is its weight when we score only the 200 questions ForecastBench gave to humans. It uses the same groups and formula, counting only the group members in the human set. "Not asked" means the question is not in the human set.

For dataset questions, both columns give the weight per question. Each of the question's forecasts gets that weight times 0.431, or times 0.464 for questions with four resolution dates.

Market questions

US midterm elections. 52 questions, 27 of them in the human set. Overall weight 0.108 each; human weight 0.172 each.

Taiwan local elections. 16 questions, 10 of them in the human set. Overall weight 0.240 each; human weight 0.314 each.

Fed decisions on October 28 and December 9. 6 questions, 3 of them in the human set. Overall weight 0.401 each; human weight 0.500 each.

US headline CPI for November. 5 questions, 2 of them in the human set. Overall weight 0.431 each; human weight 0.528 each.

Rotten Tomatoes score: The Hunger Games: Sunrise on the Reaping. 4 questions, one of them in the human set. Overall weight 0.464 each; in the human set it stands alone, so its human weight is 1.

WTI crude oil on November 3. 4 questions, 2 of them in the human set. Overall weight 0.464 each; human weight 0.528 each.

Rotten Tomatoes score: I Play Rocky. 3 questions, one of them in the human set. Overall weight 0.500 each; in the human set it stands alone, so its human weight is 1.

US core CPI for November. 3 questions, 2 of them in the human set. Overall weight 0.500 each; human weight 0.528 each.

CA-07 House race. 2 questions, none of them in the human set. Overall weight 0.528 each.

Israel-Iran ceasefire. 2 questions, all of them in the human set. Overall weight 0.528 each; human weight 0.528 each.

Nigerian presidential election. 2 questions, one of them in the human set. Overall weight 0.528 each; in the human set it stands alone, so its human weight is 1.

Spanish general election. 2 questions, all of them in the human set. Overall weight 0.528 each; human weight 0.528 each.

US CPI for October. 2 questions, none of them in the human set. Overall weight 0.528 each.

Dataset questions

Series IDs that start with ECB_ come from DBnomics. The rest come from FRED.

Euro-area government bond yields. 16 questions, 5 of them in the human set. Overall weight 0.240 each; human weight 0.431 each.

US long-term interest rates. 10 questions, 3 of them in the human set. Overall weight 0.314 each; human weight 0.500 each.

Corporate credit spreads. 7 questions, none of them in the human set. Overall weight 0.375 each.

US short-term interest rates. 7 questions, 5 of them in the human set. Overall weight 0.375 each; human weight 0.431 each.

Cleveland Fed expected inflation. 3 questions, one of them in the human set. Overall weight 0.500 each; in the human set it stands alone, so its human weight is 1.

US dollar. 3 questions, one of them in the human set. Overall weight 0.500 each; in the human set it stands alone, so its human weight is 1.

US stock indexes. 2 questions, none of them in the human set. Overall weight 0.528 each.