PDA

View Full Version : Roc area in weka explorer vs Roc area in weka experimenter



francy_faraone
11-07-2008, 06:23 AM
Does anyone know which is the relation between the value of the Roc area (AUC) resulting when using Weka Explorer, and the AUC resulting when using the Experimenter?

The Explorer show the AUC for each value of the class; First I tried to average the AUCs given, and then I tried to sum the AUCs weighted by the reference class's prevalence in the data set, but the results obtained in the Explorer and the Experimenter for the same data set and the same settings (algorithm and algorithm settings, test option: 10 folds cross validation, etc.) aren't equal. Why?
I run the Experimenter also for different numbers of repetitions (till 1000), but the still problem remains.

Thanks very much.

Mark
11-07-2008, 02:38 PM
The Experimenter reports the AUC for the first class value (unless you change this by using a little known option for the ClassifierSplitEvaluator in the "advanced" mode of the Experimenter).

You will not necessarily get the same result as the Explorer reports even when doing one run of cross-validation as the metrics are computed differently between the two. The Explorer does *not* average over folds for any of the metrics. Instead it just sums them up. E.g. for percentage correct, it is just the number of correctly classified instances expressed as a percentage of the number of instances in the data set (AUC is computed in the same way, predictions across all folds are ranked by the probability of the class label in question). The Experimenter does compute averages - so for AUC, it computes the average of the AUCs for each fold.

Hope this helps.

Cheers,
Mark.

francy_faraone
11-11-2008, 03:41 AM
Thanks a lot. :)