An IRT forecasting model: linking proper scoring rules to item response theory
Journal Title: Judgment and Decision Making - Year 2017, Vol 12, Issue 2
Abstract
This article proposes an Item Response Theoretical (IRT) forecasting model that incorporates proper scoring rules and provides evaluations of forecasters’ expertise in relation to the features of the specific questions they answer. We illustrate the model using geopolitical forecasts obtained by the Good Judgment Project (GJP) (see Mellers, Ungar, Baron, Ramos, Gurcay, Fincher, Scott, Moore, Atanasov, Swift, Murray, Stone & Tetlock, 2014). The expertise estimates from the IRT model, which take into account variation in the difficulty and discrimination power of the events, capture the underlying construct being measured and are highly correlated with the forecasters’ Brier scores. Furthermore, our expertise estimates based on the first three years of the GJP data are better predictors of both the forecasters’ fourth year Brier scores and their activity level than the overall Brier scores obtained and Merkle’s (2016) predictions, based on the same period. Lastly, we discuss the benefits of using event-characteristic information in forecasting.
Authors and Affiliations
Yuanchao Emily Bo, David V. Budescu, Charles Lewis, Philip E. Tetlock and Barbara Mellers
Keywords
Related Articles
- EP ID EP678279
- DOI -
- Views 143
- Downloads 0
Divergence between individual perceptions and objective indicators of tail risks: Evidence from floodplain residents in New York City
This study provides the first comprehensive analysis of individual perceptions of tail risks. It focuses not only on the probability, as has been studied by Nicholas Barberis and others, but also on anticipation of damag...
The persistence of common-ratio effects in multiple-play decisions
People often make more rational choices between monetary prospects when their choices will be played out many times rather than just once. For example, previous research has shown that the certainty effect and the possib...
Leftmost-digit-bias in an enumerated public sector? An experiment on citizens’ judgment of performance information
Numerical performance information is increasingly important to political decision-making in the public sector. Some have suggested that biases in citizens’ processing of numerical information can be exploited by politici...
Cross-cultural support for a link between analytic thinking and disbelief in God: Evidence from India and the United Kingdom
A substantial body of evidence suggests that favoring reason over intuition (employing an analytic cognitive style) is associated with reduced belief in God. In the current work, we address outstanding issues in this lit...
Investor regret: The role of expectation in comparing what is to what might have been
Investors, like any decision maker, feel regret when they compare the outcome of an investment with what the outcome would have been had they invested differently. We argue and show that this counterfactual comparison pr...