“What should I be testing with my email program?”

“We test but not as often or purposefully as we’d like.”

“How do we get better at email testing?”

 

These are questions and comments that often come up when we start a new engagement in email program discovery. Iterative, methodical testing is part of a solid email program foundation–and an aspect that companies often get wrong.

Spoiler alert: this article is about iterative email testing, but that doesn’t diminish the impact that something similar to a Taguchi test can have on a program. For iterative testing, I recommend a version of the agile methodology with a strong focus on ideal user experience, a process which is traditionally applied to software development.

How to Apply Agile to Email:

Obviously some of the specifics around methodologies like Scrum and Kanban aren’t going to fully translate to email testing plans, but you can borrow the framework to highlight which elements within an email should be tested based on impact and level of effort to implement. For those who may be interested, QA Symphony has a more complete write-up on the differences between Agile methodologies and testing methods. The goal is to have a transparent, fast, collaborative, and data-driven email testing process.

Step 1: Plan

There are essentially three categories for testable (or at least addressable) elements within a given campaign email: Pre-Open, Content, and Destination.

These are, with some exceptions, already ranked from left to right based on level of effort and impact. Most of the Pre-Open elements are focused on increasing open rates, while most of the Email Body elements are focused on increasing click rates. The Destination category usually requires the most amount of work outside of the email team’s direct control but shouldn’t be overlooked, as these items can also have an impact on a campaign’s overall success.

Each of these elements should be tested or addressed in every email campaign. Plan to address each item until statistical significance is reached, or at least until you’re comfortable with the directional results. It is important to call out the distinction between statistical significance and a timed winner for a particular segment of a test—the former is relevant for learnings and design guidelines, while the latter is simply choosing which variation to use for a single send.

Step 2: Design + Develop

This is where the line blurs the most when trying to borrow from the software creation process. I tend to treat Design and Develop as the same step with email testing.

Once you know the categories and elements that need to be tested for each program, you can start getting specific and writing a hypothesis for each test. It is important to standardize the hypothesis writing format so that all hypotheses have a goal and so that you know how each goal will be measured. ThoughtWorks has a great write-up with more information, but essentially you’ll want your hypothesis to look something like this:

 

We believe that ________

will result in _________.

We will know we have succeeded when ________.

 

This format requires stakeholders to think about the details of the test before handing something off for creative development.

Step 3: Test + Deploy

Email testing tools are often included in enterprise-level ESPs, but they rarely take statistical significance into account. Instead, you’re allowed to set a timeframe for the initial portion of the test, then the “winner” is sent to the holdout group based on the initial inbox metrics. This is usually good enough for email testing, but we should be mindful of factors such as site behavior that can contribute to a conversion, especially if conversions determine success for a particular test. Other factors can include list size, user activity level, and seasonality.

Step 4: Share

Avoid the tribal knowledge trap! Be sure to share your test results internally so that other email stakeholders can see what works and what doesn’t work in your email program. This can be as simple as a Google Doc or as complex as an ongoing series of blog posts. Either way, your test results should include the date and time of the test, the variables tested, the hypothesis, overall metrics by test group, and any additional notes that can help contextualize the test. Statistical significance should be clearly indicated.

Sharing results also helps prioritize future tests. Maybe a subject line element performed much better than anticipated, which could lead the team to add several more tests to zero in on what works and what doesn’t. This process can be organized into one-or two-week sprints. Overall testing results can be reviewed in quarterly business reviews or monthly meetings, but you should aim to test as often as reasonable/possible.

Conclusion

Adaptive planning, early delivery, and continuous improvement are all principles that should be applied to a structured email testing plan. Email marketers would be wise to borrow from their software developer friends here.

 

author image

Scott Burdsall