Too soon to tell if the US intelligence community prediction market is more accurate than intelligence reports: Commentary on Stastny and Lehner (2018)
Journal Title: Judgment and Decision Making - Year 2019, Vol 14, Issue 3
Abstract
Stastny and Lehner (2018) reported a study comparing the forecast accuracy of a US intelligence community prediction market (ICPM) to traditionally produced intelligence reports. Five analysts unaffiliated with the intelligence reports imputed forecasts from the reports after stating their personal forecasts on the same forecasting questions. The authors claimed that the accuracy of the ICPM was significantly greater than that of the intelligence reports and suggest this may have been due to methods that harness crowd wisdom. However, additional analyses conducted here show that the imputer’s personal forecasts, which were made individually, were as accurate as ICPM forecasts. In fact, their updated personal forecasts (made after reading the intelligence reports) were marginally more accurate than ICPM forecasts. Imputed forecasts are also strongly correlated with the imputers’ personal forecasts, casting doubt on the degree to which the imputation was in fact a reliably inter-subjective assessment of what intelligence reports implied about the forecasting questions. Alternative methods for comparing intelligence community forecasting methods are discussed.
Authors and Affiliations
David R. Mandel
Keywords
Related Articles
- EP ID EP678416
- DOI -
- Views 133
- Downloads 0
A note on determining the number of cues used in judgment analysis studies: The issue of type II error
Many judgment analysis studies employ multiple regression procedures to estimate the importance of cues. Some studies test the significance of regression coefficients in order to decide whether or not specific cues are a...
Behind the veil of ignorance: Self-serving bias in climate change negotiations
Slowing climate change will almost certainly require a reduction in greenhouse gas emissions, but agreement on who should reduce emissions by how much is difficult, in part because of the self-serving bias—the tendency t...
Are good reasoners more incest-friendly? Trait cognitive reflection predicts selective moralization in a sample of American adults
Two studies examined the relationship between individual differences in cognitive reflection (CRT) and the tendency to accord genuinely moral (non-conventional) status to a range of counter-normative acts — that is, to t...
Why are lotteries valued less? Multiple tests of a direct risk-aversion mechanism
Recent studies have identified the uncertainty effect (UE), whereby risky prospects (e.g., a binary lottery that offers either a $50 or $100 gift certificate) are valued less than their worst possible outcome (a $50 cert...
Modeling and debiasing resource saving judgments
Svenson (2011) showed that choices of one of two alternative productivity increases to save production resources (e.g., man-months) were biased. Judgments of resource savings following a speed increase from a low product...