October 2, 1998

Rob Holte
School of Information Technology and Engineering, Ottawa

ROC Convex Hulls for Comparing Machine Learning Algorithms

Machine learning algorithms are normally compared by estimating the accuracy of the classifiers they produce. However, even within the classification context there are alternative methods for comparing learning schemes that are preferable to accuracy in certain circumstances. One such method is the ROC ("receiver operating characteristic") curve. ROC curves have been used for decades in other fields; their usefulness for comparing machine learning algorithms has recently been advocated by Tom Fawcett and Foster Provost. They observe that ROC curves allow one to compare learning schemes when one is uncertain about the probability with which each class will occur during testing (which is often different than the class probabilities in the training set) and uncertain about the costs associated with different types of misclassification.

This talk assumes no prior knowledge of ROC curves. A background in machine learning is helpful but not essential to understanding the key ideas (which are useful for almost any empirical study).

Pointers to background information and a copy of the slides may be found at http://www.site.uottawa.ca/~holte/Learning/ROCtalk/index.html