A method for evaluating elicitation schemes for probabilities

Authors:

Haiqin Wang, Denver H. Dash and Marek J. Druzdzel
Decision Systems Laboratory
School of Information Sciences
and Intelligent Systems Program
University of Pittsburgh
135 North Bellefield Avenue
Pittsburgh, PA 15260, U.S.A.
e-mail: wang@sis.pitt.edu, ddash@isp.pitt.edu, marek@sis.pitt.edu

Abstract:

We present an objective approach for evaluating probability elicitation methods in probabilistic models. Our method draws on ideas from research on learning Bayesian networks: if we assume that the experts knowledge is manifested essentially as a database of records that have been collected in the course of the experts experience, and if this database of records were available to us, then the structure and parameters of the experts beliefs could be reliably constructed using techniques for Bayesian learning from data. This learned model could, in turn, be compared to elicited models to judge the effectiveness of the elicitation process. We describe a general procedure by which it is possible to capture the data corresponding to the experts beliefs, and we present a simple experiment in which we utilize this technique to compare three methods for eliciting discrete probabilities: (1) direct numerical assessment, (2) the probability wheel, and (3) the scaled probability bar. We show that for our domain, the scaled probability bar is the most effective tool for probability elicitation.