Back to the Basics: A Testing Calendar

Andrew Kordek

It seems that people have been preaching the testing thing for quite a while now and Trendline is in complete agreement with a majority of the crowd when it comes to testing. In fact, as an agency, we have seen some sizable gains in our customers KPI’s when a disciplined and doable test plan is adhered to. Once an organization gets to see positive results of creating and executing against a test plan, they literally become addicted (in a good way) to wanting to always beat the new control. However, there are some organizations that struggle on just where to start. If you are one of them… on.

A basic and effective test calendar is as hard as you want to make it. However, there are four elements that should be included in even the most basic of calendars.

A Hypothesis

The first step to building a beneficial testing calendar is clearly defining what you’re hoping to learn and achieve for each test. This will help you focus efforts and keep you on track during the testing process. When forming a hypothesis, think about things that will move the needle in a monumental vs. incremental way, and then align your objectives with these factors to formulate your hypothesis. In the early stages of hypothesis brainstorming, nothing should be off the table in terms of wanting to push the envelope to learn more about the email program itself.  

Here are a few effective vs. ineffective hypothesis gathered through the years:

Effective:  The use of emotions in SL’s over the course of 6 weeks causes greater opens AND conversions.

Ineffective: Does using less than 20 characters in the SL cause greater opens?

Effective: The sustainable use of interactive elements causes greater engagement within the email and lead to greater conversions and less attrition.

Ineffective: Does having an interactive surprise element in our promotional email lead to more revenue?

You will notice that the effective hypothesis’ are not questions, they are statements. A hypothesis is an educated, testable prediction about what will happen. Marketers tend to want questions answered and, unfortunately, that does not produce an effective email test and learn environment.

A Detailed Description

Once you have a hypothesis you need to ensure that a detailed description is written out around things like the data you’ll be using, elements/factors that would be considered, and the success metrics used to determine statistical significance. This is crucial to every stakeholder in the test, so there should be no doubt around the details. This is especially true in the case of monumental tests that would run for longer periods of time.

Effort. Impact. Timing.

One area where many organizations struggle is the ability to determine the level of effort it will take to pull off a test. Stakeholders need to determine and disseminate just how hard this venture is going to bet. For example, you can’t just say that the LOE (level of effort) to do a multi-factor creative test is going to be low, given the number of people that might/should be involved. Once the LOE is determined, the stakeholders need to make a judgment on just how big the impact to the program will be if the hypothesis is proven right/wrong. No one wants to go through a high level of effort for low impact, as it will often discourage those involved. Last but not least, realistic timing needs to be determined. We all live in a world where everyone thinks pulling off tests and learnings should be done lickety-split, but give yourself more time than you think. Especially if there are stakeholders from multiple groups involved.

Everything Else.

It goes without saying, that the rest of the testing calendar can be filled with things like dates, results, segments, comments, etc. However you look at it, your testing calendar should be like a library and everything that has been done or will be done needs to be kept in a place where everyone, including your successor, can have access. Think of your testing calendar as a  place to do historical research around what has been done in the past, so that you can build on it for the future. Instead of just saying “oh..we tried that a few years ago and it didn’t work,” you can bring up the results and see just how much “it didn’t work” and determine if it should be done again.

A testing calendar should not be a daunting task. Think of it as your own personal testing and results library. A testing calendar is like training for a triathlon, in that on day one you don’t want to run 13 miles because you can. You need to build up strength and feel the adrenaline rush of results before going all out. There is nothing more satisfying than standing up in a room full of people with results and a plan to achieve more.   


If you are looking for some help with building a testing calendar, drop us a line.



About the Author(s)

Andrew Kordek

Andrew Kordek is a Co-Founder of Trendline Interactive.

Let's Take This to the Inbox

Sign up for our news, resources and updates. The inbox is our favorite place after all. We’ll make sure it’s worth it. (You can unsubscribe at any time, but you probably already knew that.)