The Multi-Variable Test

Direct marketing offers a chance for accurate testing — something that is not really possible in any other marketing medium: TV, Radio, Print, Billboards, POP or in-store displays. Using these media, you can often notice a lift in sales, but you can never be completely sure what it was that produced the lift. On the other hand, you can use direct mail or email to send out 100,000 offers, and find out exactly how many customers make responses and purchases.  You can also use DRTV with an 800 number or a website for response. Using these you can track actual results rather than awareness, attitudes, or general increase (or decrease) in sales.

Using direct marketing, there are many things that you can test: the list, the offer, the package, the price, the season, the time of day, the creative, the paper stock, and the teaser copy. All these can be tested against controls: a control package, and a control group of customers. Putting them all together, many marketers create hundreds of test cells in Excel spreadsheets that are too large to be printed on a single sheet of legal sized paper. What a mistake!

The purpose of testing is to measure accurately the effect of a variable (such as the price) on the result (such as response or sales).  Once you add a second variable, accuracy tends to go out the window. Say you want to test two prices: $39.95 and $49.95. You have two offers: buy one or buy one with a second one at half price. You also want to test teaser copy on the envelope versus no teaser copy. You are testing 100 lists. Right away you have a complex back end analysis:

If you multiply that out, you will find that you have 1,400 test cells from this simple test. What is wrong with that? In the first place, each cell tested will either be small and inaccurate, or too large for your test budget. For accuracy, a test should have at least 500 responses. If you expect a 2% response rate, that means you have to mail out a minimum of 25,000 pieces. LetÍs say you send out 25,000 at $39.95 and 25,000 at $49.95. With 100 lists, you will be mailing 250 from each list at $39.95 and 250 at $49.95. If your expected response rate is 2% you can expect 5 responses from each price range from each list. Five expected responses are too small for accuracy. By chance, your best list may have only three responses, and another less performing list may also have only three responses. You will learn nothing. But that is not the worst of it. Since you are testing Buy One or Buy Two, each with Teaser and No Teaser, your five expected responses per list will boil down to an expectation of less than one response per cell.

No problem, you say, ñwe will increase the size of the test. Let’s send out 200,000.  If you do the math, you will see that by mailing 200,000, each of the 1,400 cells will have only 142 mailed, and an expected response rate of 2.4 sales per cell. This will not be accurate.  To get real accuracy, you will have to mail about one million. That is much larger than most test budgets.

But the mathematics is not really the problem here. The problem is that you are trying to cram too much information into a single test. The goal of a test is to learn something that you can replicate on a rollout. For example, if you sell 400 at $49.95 and 500 at $39.95, which is the best price?

Looking at revenue, it would seem that the offers are identical. For direct mail at $49.95 you have to ship only 400 products to yield $20,000. So the higher price could yield a higher profit. Many catalogers today make more money on shipping and handling than they do on product sales. For them the $39.95 could be more profitable. For database marketing, the goal of an acquisition mailing is to build a base of responsive customers who will buy from you a second and a third time. Acquiring 500 customers could be more profitable than acquiring 400 customers.

In Denny Hatch and Don JacksonÍs famous book 2,239 Tested Secrets for Direct Marketing Success, Don Nicholas said: ñThe Holy Grail of direct marketing is the single-variable test. You want only one thing to change in each test. If youÍre going to test price, then you test two packages that are the same in all respects except priceî.  This suggests that multi-variable testing, while possible, is not a good idea. Marketers today often have too much information. It is difficult or impossible to make marketing decisions if you have hundreds of pieces of information that cannot be easily grouped to decide what to do.  Ed Mayer in the book cited above said, ñSuccess in direct mailing is dependent on the following ratios: 40 percent lists, 40 percent offer, 20 percent everything else.

To do intelligent testing, your first job is to determine the most important questions which must be answered. Following Ed MayerÍs (and everyone elseÍs) rules, you must test your lists to find the lists that work best for you and your product. What is next? The offer, which usually boils down to price.

From 1994 to 2000, Paul Wang and I gave 28 two day seminars for The Database Marketing Institute. I had built up a list of 35,000 database marketers by raffling off a book at each of my hundreds of speeches and from NCDM attendees.  We mailed to this list with an average of 0.14% sales rate — which was highly profitable. We tried different prices from $1,595 to $2,195. Price did not seem to be important. No matter what we charged we still got 0.14% sales. We tested a brochure against a brochure with a personal letter. We got about nine times as many sales from the personal letter as we did from the brochure alone. Those were before the days of the Internet, so we did not test emails. What finally killed the seminars was the market crash in 2000. Our response rate fell to 0.07% which was highly unprofitable.

In testing, you are always trying to beat a control. The Wall Street Journal used a direct mail piece, ñTwo young menî for more than thirty years. Written by freelancer Martin Conroy and first mailed in 1975, it is in the form of a two page letter. No other subscription piece for the WSJ has ever been able to beat this control. In late 1991, Denny Hatch interviewed WSJ National Subscription Manger Paul Bell. According to Bell, the number of mail order subscribers to the WSJ brought in each year was approximately one million. Approximately 55% of all mail order subscribers came in as a result of the “Two Young Men” piece. This two page letter has brought in more than $1 billion to the WSJ. It is undoubtedly the most successful single piece of advertising in the history of the world.

In summary, if you are considering a multi-variable direct marketing test, think again. Figure out the most important questions that need answering, and create a series of single variable tests to get them answered.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.