Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Help about Accuracy, ROC Area, TPR, FPR, Precision Between three Classifiers

  1. #1
    Join Date
    Apr 2016
    Posts
    1

    Default Help about Accuracy, ROC Area, TPR, FPR, Precision Between three Classifiers

    I made three classifications in Weka 3.6 by using same dataset and arff file. I chose classifiers as: NaiveBayes, J48 and SMO. I have a problem/misunderstanding about outputs.
    Bayes had the lowest accuracy but highest ROC area, J48 had lower ROC Area than Bayes but highest ACC. Are not suppose to be the classifier which have highest ACC also have highest ROC Area?

    Secondly, Bayes had the lowest TP Rate but also had the lowest FP Rate and highest Presicion for the class b. In addition SMO had the highest TPR for class b and also had the highest FPR. How can it happen? Could you please help me about this results? My classifier outputs are below:

    Bayes:
    === Stratified cross-validation ===
    === Summary ===


    Correctly Classified Instances 661 91.8056 %
    Incorrectly Classified Instances 59 8.1944 %
    Kappa statistic 0.8361
    Mean absolute error 0.0881
    Root mean squared error 0.2805
    Relative absolute error 17.6262 %
    Root relative squared error 56.0991 %
    Total Number of Instances 720


    === Detailed Accuracy By Class ===


    TP Rate FP Rate Precision Recall F-Measure ROC Area Class
    0.994 0.158 0.863 0.994 0.924 0.978 a
    0.842 0.006 0.993 0.842 0.911 0.978 b
    Weighted Avg. 0.918 0.082 0.928 0.918 0.918 0.978


    J48:
    === Stratified cross-validation ===
    === Summary ===


    Correctly Classified Instances 683 94.8611 %
    Incorrectly Classified Instances 37 5.1389 %
    Kappa statistic 0.8972
    Mean absolute error 0.0746
    Root mean squared error 0.2165
    Relative absolute error 14.9276 %
    Root relative squared error 43.2941 %
    Total Number of Instances 720


    === Detailed Accuracy By Class ===


    TP Rate FP Rate Precision Recall F-Measure ROC Area Class
    0.975 0.078 0.926 0.975 0.95 0.973 a
    0.922 0.025 0.974 0.922 0.947 0.973 b
    Weighted Avg. 0.949 0.051 0.95 0.949 0.949 0.973

    SMO:
    === Stratified cross-validation ===
    === Summary ===


    Correctly Classified Instances 680 94.4444 %
    Incorrectly Classified Instances 40 5.5556 %
    Kappa statistic 0.8889
    Mean absolute error 0.0556
    Root mean squared error 0.2357
    Relative absolute error 11.1111 %
    Root relative squared error 47.1405 %
    Total Number of Instances 720


    === Detailed Accuracy By Class ===


    TP Rate FP Rate Precision Recall F-Measure ROC Area Class
    0.953 0.064 0.937 0.953 0.945 0.944 a
    0.936 0.047 0.952 0.936 0.944 0.944 b
    Weighted Avg. 0.944 0.056 0.945 0.944 0.944 0.944

  2. #2
    Join Date
    Aug 2006
    Posts
    1,741

    Default

    AUC is a measure that summarises the ROC curve. The higher the AUC, the better the ranking performance of the classifier. It can be interpreted as the probability that a randomly selected positive instance will be ranked higher (according to the predicted probability of being positive) than a randomly selected negative instance. Accuracy corresponds to just one point on the ROC curve/ranked list of predictions, i.e the point on the curve/list that corresponds to a threshold of 0.5 on the predicted probability of the positive class.

    Take a look at the wikipedia article on ROC:

    https://en.wikipedia.org/wiki/Receiv...characteristic

    Cheers,
    Mark.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.