Articles - Written by Arthur Hughes - 4 Comments
Determining the Minimum Test Size
Most people come to database marketing from some other discipline: advertising, general marketing, market research, statistics, customer communications, data processing. Only a few have a solid background in direct marketing. That is too bad, because many significant problems that are confronted by database marketers today are the same ones that were dealt with and solved more than twenty years ago by direct marketers. As an industry we have an inadequate grasp of our history. It is amazing to discover how many professionals in database marketing do not understand RFM or Lifetime Value – to take two examples. Those who do not remember the past are condemned to repeat it.
Nothing illustrates this problem better than the issue of tests and controls. When I bring up a word like “Controls” to a class of modern database marketing professionals, half of them draw a blank. A Control, of course, is a temporary standard: either an existing marketing package against which you are testing a new marketing package, or a group of customers who are similar to a group being tested, but which do not receive a promotion given to the test group. You use the controls to measure the effectiveness of the test.
A Preferred Customer Card
A couple of years ago, an advertising agency recommended that a major financial services company issue preferred customer cards to everyone who had used their services three or more times. Special benefits were given to these card holders. The benefits were not available to other customers. The program was supposed to build loyalty and increase usage. Perhaps it did. But three years later, the client asked the agency to come up with some proof as to what value they had obtained for the millions of dollars that had been expended on the preferred customer program. The agency came to us for advice.
- “Where are the controls?” I asked.
- “What do you mean by controls?”
- “Controls are the people who used the service three or more times who were not given the preferred customer cards. You measure your success by comparing the retention and spending patterns of the controls against those who did receive the cards.”
- “There were no such people. We gave the cards to everyone who deserved them.”
- “Then you are out of luck. I don’t know how you can prove that your preferred customer cards did you any good.”
The agency lost the contract. The problem here is that the theory of controls is taught in Direct Marketing 101. It is a concept so fundamental that it is embarrassing to have to bring it up to a marketing professional. But, let’s go on.
Not every database marketing strategy is successful. Some of them are costly failures. Others are only marginally successful in their basic goals which are: building loyalty, increasing sales, stimulating referrals, or reducing the cost of acquisition. To know which of the tactics, lists, packages, methods, prices, offers, or rewards you have decided to experiment with is working effectively, you have to conduct tests, and you should also set aside controls. Most people would agree with this reasoning. Where you get a disagreement is over this question: “How large should the test and control groups be to provide you with a reasonable assurance of scientific accuracy?”
Two Conflicting Objectives
There are two conflicting objectives at work here. You want to have your test samples as large as possible so as to be sure of scientific accuracy. At the same time, you want your test samples as small as possible so as to keep the cost of your test to a minimum. Test budgets are usually tight. What is the ideal minimum size that satisfies both objectives? In the balance of this article, we are going to go into the details of determining the ideal minimum test cell size. The methods outlined here are available to everyone reading this page. You don’t need a statistician. You don’t need a model. You can work it out on a hand calculator.
Break Even Rate
Let’s begin by assuming that we are going to send some promotion to our customer base, or to a group of prospects. We could be using direct mail, fax, e-mail or outbound telemarketing. We have several approaches that we want to test. We will set up a group of customers, called cells, which we will use to test each approach. The approaches could be lists, packages, offers, RFM cells. For each promoted cell, we will determine the response rate (RR). This rate, of course, is calculated by dividing the number of responders (buyers) by the number promoted (mailed or telephoned). Let us assume that we directed our promotion at thirty thousand households and 386 of them accepted the offer. The response rate is:
RR = (# Respondents) / (# Promoted)RR = 386 / 30,000 = 1.287%
Is this an acceptable response rate? That is determined by whether we made or lost money. What is our return on investment? The return on investment (ROI) is calculated by dividing the net revenue by the cost of promotion. The net revenue is the gross revenue from sales, less the cost of the product, the cost of fulfillment, the returns, and processing fees. If the average net revenue from each sale to these 386 households was $50, then the net revenue from the promotion was $19,300 (386 • $50). Let us say that it cost us $0.62 each to send the promotion to these 30,000 people. The $0.62 includes the creative, the list, the printing, personalization, postage, mailshop, Altogether, the cost of promotion is $18,600 ($0.62 • 30,000). Our return on investment looks like this:
ROI = (Net Revenue) / (cost of promotion)ROI = ($19,300 / $18,600) = 1.038
A return on investment of 1.0 represents a situation in which you just recover your promotion costs from the net revenue, in other words you just break even. In this case, we did slightly better than break even. We earned 3.8% on our original investment of $18,600. The Break Even Response Rate (BE), therefore, is that response rate when the ROI is 1.0 — the net revenue just exactly equals the cost of promotion.
Using the formula developed above, we can calculate – in advance – what level of response we would need from any promotion to just break even. The formula is:
BE is when (Net Revenue) = (Cost of Promotion)Net Revenue = (Number of Responses) • (Average Net Revenue Per Response)NR = Resp • NetRevCost of Promotion = (Number Promoted) • (Average cost per outgoing piece)CP = Prom • PPCostBE is when (Resp) • (NetRev) = (Prom) • (PPCost)Solving this formula for Resp: Resp = ((Prom • (PPCost)) / (NetRev)When you substitute Resp in the response rate formula, you end up with the formula:BE = (PPCost) / (NetRev)
This formula tells us the break even response rate for any combination of outgoing per piece costs divided by any combination of average net revenue from successful sales. For example, if the per piece cost of promotion is $0.62 and the net revenue from the average sale is $50, then the Break Even Rate is 1.24%. You have to get more than 1.24% successful responses from this promotion to do better than break even. Using this formula, here is the type of results you can get by sending a promotion to a group of customers divided into 125 RFM cells:
As you can see, 34 of these cells made money – did better than break even. The balance of the cells lost money. This is powerful information. If the cells we selected for the test were representative of hundreds of thousands or millions of households that were not included in the test – but shared the characteristics of those who were promoted – then we can know, in advance, how those unpromoted people would have responded to this particular offer. We can save ourselves money by skipping the losers on our rollout promotion, and by contacting those in the responsive cells twice or even three times.
How Many in Each Cell
This brings us back to the question we started with. How many must we promote in each cell to be sure that the results we get accurately predict how other similar customers will react to a similar offer? There is a simple formula which answers this question. It is this:
Minimum Cell Size = 4 / BE
In other words, if the break even rate is 1.24% (as above) then the formula is:
Min = 4 / BE = 4 / .0124 = 323 customers.
Where does the constant 4.0 come from? It is a rule of thumb. It is an average that works. In your particular situation, you may find that you can get accurate results with 3.8 or even 3.5. Others may require 4.2 or 4.5 to assure themselves of accuracy. But everyone’s number will hover somewhere around 4.0.
Why it works
For those who don’t like algebra, let’s explain in layman’s language why the minimum test size is dependent on the break even response rate. The reasoning goes this way:
Suppose that you were in a situation in which to break even you had to get a 100% response rate. If even one person turned you down, you would lose money. How many would you need to contact to determine whether you were getting a 100% response rate? Maybe 10 or 20? Certainly not too many. One defection, and you know you are in trouble.
On the other hand, suppose you could break even if only one person in a thousand accepted your offer. How many would you need to contact to be sure of that response rate? Clearly you would have to contact well over one thousand. If you contacted only one thousand and the one person who you were counting on to respond was sick that day or on vacation, then you would assume that your offer was no good – when it might have been perfectly good, but you were defeated by mere chance. To keep chance from gumming up your test results, you must contact enough. In this case, the formula suggests that you must contact 4,000 to assure yourself that you truly have a response rate of 1 per thousand.
Let’s Hear From You
As you can see by the foregoing, there is a lot of solid reasoning that lies behind successful database marketing. We learn by sharing knowledge and experience. I am sure that many of the readers of this article have experience with Tests and Controls, Return on Investment, Break Even Response Rates, and Minimum Test Sizes. Your experience and logical framework may differ from the approach suggested here. Let’s get a dialog going. If you have a different – and perhaps better – way of dealing with these concepts, I would love to hear from you. I invite you to write, fax, e-mail or phone me. I will publish the differing concepts in a future article.
Arthur Middleton Hughes, vice president of The Database Marketing Institute, has presented 28 seminars on database and email marketing. Arthur has also authored several books includingStrategic Database Marketing 4th Edition (McGraw-Hill 2012). He and Andrew Kordek, chief strategist and co-founder of Trendline Interactive, are hosting a two-day Email Strategy Study Group in Fort Lauderdale March 26-27, 2013, featuring group competition for email marketers responsible for subscriber acquisition, lifetime value, ratings and reviews, boosting their email budget, and doubling their ROI. To learn how to attend the Study Group,click here
Post Footer automatically generated by Add Post Footer Plugin for wordpress.
Featured Books
Strategic Database Marketing 4th Edition: from Arthur Hughes.
To take the quizes from this book click here.
Successful E-mail Marketing Strategies: from Hunting to Farming (RACOM 2009) by Arthur Middleton Hughes and Arthur Sweetser.
This new book takes readers through a step by step approach to implementing database marketing essentials to improve e-mails marketing returns intelligently. This is the book that the trade press has been raving about.
The two Arthurs make it easy to see the difference between primitive “Hunting” and customer-centric “Farming”. This is a game-changing attitudinal shift. If you want your e-mail marketing to succeed, learn how to farm.—Stephanie Miller, VP Return Path Click here to download free the exhibits from this new book.
Strategic Database Marketing: from Arthur Hughes.
Customer Churn Reduction and Retention for Telecoms: Models for All Marketers: from Arthur Middleton Hughes.
Arthur Middleton Hughes has just published a new book of Marketing Strategy for Telecoms including Landline and Wireless Telcos, Cable TV, Satellite TV, BPL, IPTV, VoIP and WiMAX.
More In Articles
- How to Do Your Own Market Research & Identify Your Ideal Customer
- Email Strategy Study Group
- Why 21 Links Could Be Key to Your Email Success
- Why do e-mails bounce?
- How does frequency of e-mails affect open, click and conversion rates?
More In Telecom Articles
- Glossary of Telecom Marketing Terms
- Telecom Marketing Issue 01
- Creative Destruction Hits Telecom
- Case Study: Wireless Churn Reduction
- HDTV Bandwidth Requirements
More In Speeches
- Targeted Addressable Advertising Using Digital TV
- Speech on Distance Selling delivered in Athens, Greece by Arthur Middleton Hughes on December 15 2009
- Speech by Arthur Middleton Hughes on How to Retain Insurance Customers Delivered in Philadelphia to PIMA on November 17 2009
- Surviving the Recession Surviving the Recession with E-mail Marketing
- Online Marketing: Driving Traffic, Conversion and Sales
