A/B Testing Email: Critical to Your Strategy

Cross-posted from USA.gov

USAGov's goal is to improve people’s lives by connecting them to trusted government information and services in English and Spanish. One approach we use to inform the public about government information and to promote our services is through email. Consumers’ inboxes are crowded; people check their email more than 74 times a day, on average. So we use A/B testing to ensure our information is clear and engages our readers.

What is A/B Testing?

A/B testing, also known as split testing, is a method of testing through which variables are compared to each other to identify the one that brings a better response rate. In email, the element that is being tested is called “control” and the element that is argued to give a better result is called “treatment.” Running A/B tests in your outreach efforts is a great way to learn how to drive more traffic to your website and generate more actions from your audience.

As we've observed, just a few small tweaks to an email's subject line or call-to-action can significantly affect engagement numbers. Common elements that can be A/B tested in email are the format, layout, timing, sender address and subject lines. Though minor copy changes or general improvements don’t always give conclusive information or sway overall engagement numbers, this method of testing allows for understanding preferences of why, how and where different audiences receive information.

Best Practices of A/B Testing

• Create Clear and Concise Subject Lines: Subject lines that drive email opens tend to be less than 50 characters in length, contain a clear call to action, and inform the reader what to expect in the email.
• Test Both Major and Minor Changes: Although it’s reasonable to think that big, sweeping changes can increase your click numbers, the small details are often just as critical. While creating your tests, remember that even a simple change, like switching the image in your email, can drive big improvements. In fact, these sorts of changes are usually easier to measure than the bigger ones.
• Always Set Up Control and Treatment: In any experiment, you need to keep a version of the original element you’re testing. When conducting A/B tests, set up your unaltered version as your “control,” the email you would normally use. From there, build variations, or “treatments,” the emails you’ll test against your control. This will ensure always knowing your baseline for measurement.
• Send the Winning Email: After the email has been sent to a small yet statistically significant group of recipients, you'll most likely have a clear winner, email A or email B. Send the winning email to the larger audience.

Ultimately, a great email will contain a clear subject line followed by compelling copy, images that bring the reader in, and calls-to-action that resonate with what the reader is looking for and connects with. By constantly testing and utilizing email marketing best practices, we aim to effectively communicate important public service messages. Our high open and much improved click rates are testaments to this testing process.

In conclusion, while there are best practices to follow, you'd be surprised by what results your tests render. Part of testing elements of your email are testing your assumptions about how people consume your content and what they respond to. Does a shorter subject line in fact get more opens? Take the guesswork out of your email marketing strategy by testing, evaluating, and then delivering the best email you can.