A method for evaluating elicitation schemes for probabilistic models

Authors:

Haiqin Wang, Denver H. Dash and Marek J. Druzdzel
Decision Systems Laboratory
School of Information Sciences
and Intelligent Systems Program
University of Pittsburgh
135 North Bellefield Avenue
Pittsburgh, PA 15260, U.S.A.
e-mail: wang@sis.pitt.edu, marek@sis.pitt.edu, ddash@isp.pitt.edu

Abstract:

We present an objective approach for evaluating probability and structure elicitation methods in probabilistic models. The main idea is to use the model derived from the experts experience rather than the true model as the standard to compare the elicited model. We describe a general procedure by which it is possible to capture the data corresponding to the experts beliefs, and we present a simple experiment in which we utilize this technique to compare three methods for eliciting discrete probabilities: 1) direct numerical assessment, 2) the probability wheel, and 3) the scaled probability bar. We show that for our domain, the scaled probability bar is the most effective tool for probability elicitation. Index TermsBayesian network, evaluation of elicitation methods, learning, probability elicitation.