Comparative evaluation of the forecast accuracy of analysis reports and a prediction market
Journal Title: Judgment and Decision Making - Year 2018, Vol 13, Issue 2
Abstract
This paper summarizes an empirical comparison of the accuracy of forecasts included in analysis reports developed by professional intelligence analysts to comparable forecasts in a prediction market that has broad participation from across an intelligence community. To compare forecast accuracy, 99 event forecasts were extracted from qualitative descriptions found in 41 analysis reports and posted on the prediction market. Quantitative probabilities were imputed from the qualitative forecasts by asking seasoned professional analysts, who did not participate in the prediction market, to read the reports and to infer a quantitative probability based on what was written. These readers were also asked to provide their personal probabilities before and after reading the reports. There were two statistically significant results of particular interest. First, the primary result is that the prediction market forecasts were more accurate than the analysis reports. On average prediction market probabilities were 0.114 closer to ground truth than the analysis report probabilities. Second, in cases where analysts (readers) updated their personal probabilities in a direction opposite to what the reports implied, analysts tended to update their probabilities in the correct direction. This occurred even though, on average, reading the reports did not make readers more accurate.
Authors and Affiliations
Bradley J. Stastny and Paul E. Lehner
Keywords
Related Articles
- EP ID EP678337
- DOI -
- Views 133
- Downloads 0
The tyranny of choice: a cross-cultural investigation of maximizing-satisficing effects on well-being
The present research investigated the relationship between individual differences in maximizing versus satisficing (i.e., seeking to make the single best choice, rather than a choice that is merely good enough) and well-...
Is a picture worth a thousand words? The interaction of visual display and attribute representation in attenuating framing bias
The attribute framing bias is a well-established phenomenon, in which an object or an event is evaluated more favorably when presented in a positive frame such as “the half full glass” than when presented in the compleme...
Savings, subgoals, and reference points
Decision makers often save money for a specific goal by forgoing discretionary consumption and instead putting the money toward the savings goal. We hypothesized that reference points can be exploited to enhance this typ...
Introduction to the special issue: Coherence and correspondence in judgment and decision making
In 2007, Kenneth Hammond published Beyond Rationality: The search for wisdom in a troubled time. In this book and in prior works, Hammond (1990; 1996) promotes the recognition and equal acceptance of two different classe...
Belief in the unstructured interview: The persistence of an illusion
Unstructured interviews are a ubiquitous tool for making screening decisions despite a vast literature suggesting that they have little validity. We sought to establish reasons why people might persist in the illusion th...