I still remember my first seminar on modeling. I was shown how, with proper statistical techniques, done by a Ph.D. Statistician, one could find the top 20% of the customers who produced 80% of the profits in a mailing. Neiman Marcus ran the tests and the graphics were impressive. However, questions arose in my mind, I raised my hand. "You show very dramatic improvements over your control mailing, what customer segmentation method did you use for comparison?"
"A random sample." was the brusk reply. The answer, coming from a Dr. of Statistics probably went right by most of the audience. I knew, on the other hand, that just selecting the most recent 0-3 month buyers would probably generate similar if not better results. Adding segments of 3-6 and 6-12 would have ruined the beautiful presentation.
And those many years ago, I learned a very valuable lesson. It isn't just about testing, it is about test design and integrity. Direct Marketing does offer the possibility of learning. But it also offers the opportunity for manipulation and statistical deception. Next time you are listening to a public presentation about the magic of statistics or database segmentation or offer personalization, remember, if the numbers are detailed the client probably isn't present, if they are not detailed the testing was probably not valid.
We have been told for decades about how we can make money with data - the truth is, it is not such a simple truth.
An excellent article by George Giles on the problems with digital data Lies, Damned Lies and Statistics