riverculture

02-11-2009, 01:00 AM

Hello:

Could I please have some advice on the evalution of a model?

I have read through the formula about the following error measurements, and trying to understand the interpretation.

Mean absolute error

Root mean squared error

Relative absolute error

Root relative squared error

Is there a rule of thumb that what number is indicating the model is useful? is there a boundary or something like that? say for an arbitrary instance, something like-- if Mean Absolute Error <0.5 is good, otherwise bad? I don't really know how to interpret this 2 errors,

Mean absolute error

Root mean squared error

I know that the errors with the word 'relative' are comparing difference between the actual and predicted value and the difference between the actual value and the mean of the actual value. So if they are >=1, that means the model's predicting ability is not better than the mean's. Am I right? If my understanding is correct then, if <1, that means the model precdicts better than the mean, but how much better is good enough? Can I use the model if the relative errors are 0.8? how about 0.7 etc....

Another question if I could, the errors measurement makes sense to me when the class is numeric, but why use these error measurements for the nominal classes? (ie, what is the difference between yes and no??) the following is an example from the weather data bundled with WEKA. Or did I apply it to the wrong learning scheme?

I know it's a long question, Thank you so so much!

Wen

=== Run information ===

Scheme: weka.classifiers.trees.J48 -C 0.25 -M 2

Relation: weather

Instances: 14

Attributes: 5

outlook

temperature

humidity

windy

play

Test mode: 10-fold cross-validation

=== Classifier model (full training set) ===

J48 pruned tree

------------------

outlook = sunny

| humidity <= 75: yes (2.0)

| humidity > 75: no (3.0)

outlook = overcast: yes (4.0)

outlook = rainy

| windy = TRUE: no (2.0)

| windy = FALSE: yes (3.0)

Number of Leaves : 5

Size of the tree : 8

Time taken to build model: 0.05 seconds

=== Stratified cross-validation ===

=== Summary ===

Correctly Classified Instances 9 64.2857 %

Incorrectly Classified Instances 5 35.7143 %

Kappa statistic 0.186

Mean absolute error 0.2857

Root mean squared error 0.4818

Relative absolute error 60 %

Root relative squared error 97.6586 %

Total Number of Instances 14

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure Class

0.778 0.6 0.7 0.778 0.737 yes

0.4 0.222 0.5 0.4 0.444 no

=== Confusion Matrix ===

a b <-- classified as

7 2 | a = yes

3 2 | b = no

Could I please have some advice on the evalution of a model?

I have read through the formula about the following error measurements, and trying to understand the interpretation.

Mean absolute error

Root mean squared error

Relative absolute error

Root relative squared error

Is there a rule of thumb that what number is indicating the model is useful? is there a boundary or something like that? say for an arbitrary instance, something like-- if Mean Absolute Error <0.5 is good, otherwise bad? I don't really know how to interpret this 2 errors,

Mean absolute error

Root mean squared error

I know that the errors with the word 'relative' are comparing difference between the actual and predicted value and the difference between the actual value and the mean of the actual value. So if they are >=1, that means the model's predicting ability is not better than the mean's. Am I right? If my understanding is correct then, if <1, that means the model precdicts better than the mean, but how much better is good enough? Can I use the model if the relative errors are 0.8? how about 0.7 etc....

Another question if I could, the errors measurement makes sense to me when the class is numeric, but why use these error measurements for the nominal classes? (ie, what is the difference between yes and no??) the following is an example from the weather data bundled with WEKA. Or did I apply it to the wrong learning scheme?

I know it's a long question, Thank you so so much!

Wen

=== Run information ===

Scheme: weka.classifiers.trees.J48 -C 0.25 -M 2

Relation: weather

Instances: 14

Attributes: 5

outlook

temperature

humidity

windy

play

Test mode: 10-fold cross-validation

=== Classifier model (full training set) ===

J48 pruned tree

------------------

outlook = sunny

| humidity <= 75: yes (2.0)

| humidity > 75: no (3.0)

outlook = overcast: yes (4.0)

outlook = rainy

| windy = TRUE: no (2.0)

| windy = FALSE: yes (3.0)

Number of Leaves : 5

Size of the tree : 8

Time taken to build model: 0.05 seconds

=== Stratified cross-validation ===

=== Summary ===

Correctly Classified Instances 9 64.2857 %

Incorrectly Classified Instances 5 35.7143 %

Kappa statistic 0.186

Mean absolute error 0.2857

Root mean squared error 0.4818

Relative absolute error 60 %

Root relative squared error 97.6586 %

Total Number of Instances 14

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure Class

0.778 0.6 0.7 0.778 0.737 yes

0.4 0.222 0.5 0.4 0.444 no

=== Confusion Matrix ===

a b <-- classified as

7 2 | a = yes

3 2 | b = no