Weighted Brier score decompositions for topically heterogenous forecasting tournaments
Journal Title: Judgment and Decision Making - Year 2018, Vol 13, Issue 2
Abstract
Brier score decompositions, including those attributed to Murphy and to Yates, provide popular metrics for estimating forecast performance attributes like calibration and discrimination. However, the decompositions are generally limited to situations where forecasters make successive forecast judgments against the same class of substantive event (e.g., rain vs. no rain). They do not readily translate to common situations where: forecasts are weighted unequally; forecasts can be made against a range of heterogeneous topics and events over varying time horizons; forecasts can be updated over time until an event occurs or an event deadline is reached; or outcome alternatives can vary in number and nature (e.g., ordered vs. unordered outcomes) across forecast questions. In this paper, we propose extensions of the Murphy and Yates decompositions to address these features. The extensions involve new analytic expressions for the decompositions of weighted Brier scores, along with proposed resampling methods. We use data from a recent forecasting tournament to illustrate the methods.
Authors and Affiliations
Edgar C. Merkle and Robert Hartman
Keywords
Related Articles
- EP ID EP678336
- DOI -
- Views 170
- Downloads 0
Methodological notes on model comparisons and strategy classification: A falsificationist proposition
Taking a falsificationist perspective, the present paper identifies two major shortcomings of existing approaches to comparative model evaluations in general and strategy classifications in particular. These are (1) fail...
Lay attitudes to trade with low-wage countries
Three studies presented scenarios to lay people to investigate their willingness to restrict imports. Greater restriction was preferred when similar goods were made at home, when the owners of the foreign businesses made...
Do we de-bias ourselves?: The impact of repeated presentation on the bat-and-ball problem
The notorious bat-and-ball problem has long been used to demonstrate that people are easily biased by their intuitions. In this paper we test the robustness of biased responding by examining how it is affected by repeate...
Recognition-based judgments and decisions: What we have learned (so far)
This special issue on recognition processes in inferential decision making represents an adversarial collaboration among the three guest editors. This introductory article to the special issue’s third and final part come...
Information asymmetry in decision from description versus decision from experience
In this paper we investigate the claim that decisions from experience (in which the features of lotteries are learned through a sampling process) differ from decisions from description (in which features of lotteries are...