Test Expectations vs. Reality

Why didn’t the test perform as expected?

Sometimes marketers are confident that a marketing campaign’s test versus control analysis will turn out a certain way, only to be surprised by the results. The number of possible reasons for things turning out differently than hypothesized is quite large. A handful of them are presented below:

• Was the sample size too small for the test?
• Did other differences occur between the versions, thus causing the test to truly not be single variable (e.g., different mailing dates)?
• Did other marketing campaigns get executed during the attribution window?
• Was the call to action too difficult (e.g., did an email campaign require too many clicks and forms to be completed)?
• Was the campaign executed during the wrong season, around a holiday or during the wrong time in many prospects’ or customers’ journey paths?
• Was the response window too short (e.g., coupon only good for a couple of days)?
• Did the competition change their strategy or launch a campaign right before?
• Did a sudden change in the economy occur that strongly impacted your target audience?
• Have only a few impressions been made? Often, prospects must be hit 6+ times before a decision is made.
• If the campaign is digital, was the design and functionality tested across multiple devices and platforms?
• Are responses being collected correctly or could there be an issue where attribution is being measured incorrectly?
• Were orders canceled due to inventory or systems issues?
• Is the marketing team not aware of possible demographic shifts or out of touch with recent trends? Maybe the creative doesn’t really apply well to the audience targeted.
• Is the offer simply not compelling enough?
• Is the underlying data old, dirty or not standardized? Is it properly deduped?

And if things do go as hypothesized, after reading the above, you may now be wondering if a test versus control should be validated more than once. I believe a case could be made for exactly that. Implementing data hygiene practices and following proper marketing guidelines can be costly and time intensive, but the consequences of not doing so can be even more costly.