The acceptable real life accuracy of Market Prediction

[perryrico]

Hi Guys,

I had developed a model to predict the likelihood of customer to respond positive or negative to a marketing campaign using J48, LMT, Logistic, and Neural Networks using Cross Validation with Bagging Optimization. I manage to obtain 92% Correctly Classified Instance with Kappa Statistics approximately 80%.

However--when I used all the model in real world the scenario had predicted 72% accurate call.

1) Why is it I only got 72% accurate as compared with my 92% Correct Classified Instance and 80% Kappa Statistics?

2) Based on experience in the industry guys is this acceptable?

3) Is there other way to improve my prediction?

[Mark]

Hi Perry,

One possibility is that the historical training data is not completely representative of the new data. I.e. there has been some degree of shift in the underlying distribution of the data. This is sometimes referred to as concept drift. Given the recent, relatively rapid shift in the state of the world's economies, there is good chance that you are seeing this sort of effect

It is hard to say whether 72% is good or bad. And it really depends on your application and the costs involved. In a balanced two class problem, 72% is substantially better than random prediction. However, your application has a skewed class distribution. What is the precision and recall for the positive class? Is this a direct marketing-type of application, and are there costs involved with producing the marketing (i.e. mail-out/distribution costs)? If so, then rather than looking at accuracy (which assumes a default threshold of 0.5 on the probability of predicting the positive class), it is better to look at ROC or lift curves. On these curves you can consider different scenarios for targeted marketing, based on contacting different sized subsets of the target population (which corresponds to choosing probability thresholds other than 0.5) and the costs involved.

Cheers,
Mark.