A response to Mandel’s (2019) commentary on Stastny and Lehner (2018)
Journal Title: Judgment and Decision Making - Year 2019, Vol 14, Issue 5
Abstract
Stastny and Lehner (2018) compared the accuracy of forecasts in an intelligence community prediction market to comparable forecasts in analysis reports prepared by groups of professional intelligence analysts. To obtain quantitative probabilities from the analysis reports experienced analysts were asked to read the reports and state what probability they thought the reports implied for each forecast question. These were called imputed probabilities. Stastny and Lehner found that the prediction market was more accurate than the imputed probabilities and concluded that this was evidence that the prediction market was more accurate than the analysis reports. In a commentary, Mandel (2019) took exception to this interpretation. In a re-analysis of the data, Mandel found a very strong correlation between readers’ personal and imputed probabilities. From this Mandel builds a case that the imputed probabilities are little more than a reflection of the readers’ personal views; that they do not fairly reflect the contents of the analysis reports; and therefore, any accuracy results are spurious. This paper argues two points. First, the high correlation between imputed and personal probabilities was not evidence of substantial imputation bias. Rather it was the natural by-product of the fact that the imputed and personal probabilities were both forecasts of the same events. An additional analysis shows a much lower level of imputation bias that is consistent with the original results and interpretation. Second, the focus of Stastny and Lehner (2018) was on the reports as understood by readers. In this context, even if there was substantial imputation bias it would not invalidate accuracy results; it would instead provide a possible causal explanation of those results.
Authors and Affiliations
Paul Lehner and Bradley Stastny
Keywords
Related Articles
- EP ID EP678442
- DOI -
- Views 139
- Downloads 0
Do we de-bias ourselves?: The impact of repeated presentation on the bat-and-ball problem
The notorious bat-and-ball problem has long been used to demonstrate that people are easily biased by their intuitions. In this paper we test the robustness of biased responding by examining how it is affected by repeate...
The multiplicity of emotions: A framework of emotional functions in decision making
A four-fold classification of emotions with respect to their functions in decision making is proposed. It is argued that emotions are not homogenous concerning their role in decision making, but that four distinct functi...
The empirical content of theories in judgment and decision making: Shortcomings and remedies
According to Karl Popper, we can tell good theories from poor ones by assessing their empirical content (empirischer Gehalt), which basically reflects how much information they convey concerning the world. “The empirical...
Threshold models of recognition and the recognition heuristic
According to the recognition heuristic (RH) theory, decisions follow the recognition principle: Given a high validity of the recognition cue, people should prefer recognized choice options compared to unrecognized ones....
Bullshit for you; transcendence for me. A commentary on “On the reception and detection of pseudo-profound bullshit”
I raise a methodological concern regarding the study performed by Pennycook, Cheyne, Barr, Koehler & Fugelsang (2015), in which they used randomly generated, but syntactically correct, statements that were rated for prof...