If you're doing a startup, you need to be doing A/B testing in almost everything that you do. That's it, hands down, end of discussion.

But, why's it so important? Well, as a startup, you generally have three things you need to do:

1. Build your money-making machine
2. Learn
3. Grow

Now, obviously I've oversimplified it there for you, but that really is the main strategy of a startup.

And, to do that, you need to test hypotheses, and then learn from your results what works. Well, that's where A/B testing comes in.

What Is A/B Testing?

A/B testing (also called split testing) is a method used to compare two versions of a single variable by changing only one element. The idea is simple: test version A against version B and see which one performs better. Whether it’s adjusting website copy, sending emails, refining app features, or tweaking your pricing strategy, the goal is to discover what resonates with users and drives better results.

Let’s say you’re developing a new product feature. Instead of launching it to your entire audience, you can A/B test it with a small group. One group gets the new feature, the other group doesn’t. You then measure key metrics (like user engagement, conversion rates, or retention) and determine which version works best.

Why A/B Testing Is Crucial for Lean Startups

A/B testing is especially relevant for startups embracing the Lean Startup Methodology, which focuses on iterating quickly, learning from customers, and validating business ideas with minimal risk. In the lean framework, making assumptions and testing them in the real world is vital. A/B testing allows startups to run these experiments in a structured way.

Instead of over-committing resources to a new idea or feature, A/B testing lets you validate hypotheses early. Want to know if a 10% price hike will deter customers? A/B test it. Curious whether a feature is more valuable than it seems? Test it with a subset of users before rolling it out. These small, controlled experiments provide insights that can save startups from costly mistakes while steering them toward optimal growth strategies.

What Should You Test?

Anything that impacts user behavior can (and should) be tested. Here are some common areas startups typically explore:

• Website design & Landing Pages: Test different layouts, button colors, and even the wording of calls-to-action.
• Emails: Emails can help you retain customers. If your emails are better, more customers come back. See below for my example.
• Pricing models: Experiment with different subscription plans or one-time payment options.
• Product features: Assess the impact of new features on user engagement or retention.
• Marketing strategies: Test variations of ad copy, landing pages, or even social media campaigns to see what drives the most conversions.
• Basically everything: Yes. Yes you should.

How to A/B test Emails

A/B testing in emails allows you to experiment with different elements, such as subject lines, CTA buttons, or even the layout, to see which version resonates best with your audience. 

Most email service providers (ESPs) provide A/B testing functionality out of the box. If you're in E-commerce, I'd recommend to use Klaviyo, it's awesome for stuff like this. Actually, on second though, if your sending any type of email marketing....just use Klaviyo ;)

Ha, just kidding, if you're doing email newsletters, then Beehiiv has some more specific functionality. But yeah, at the time of writing this (18/10/2024), Klaviyo still is better for A/B testing. 

Anyways, here's a list of stuff you can A/B test within your emails:

• Subject line: Length, wording, personalization, or use of emojis.
• Call-to-action (CTA): Button color, size, placement, or text.
• Email layout: Single-column vs. multi-column, text vs. image-heavy.
• Send time: Different days of the week or times of day.
• Preview text: The snippet that appears next to the subject line.
• Images: Use of images vs. no images, static vs. animated (GIFs).
• Email copy: Short vs. long, formal vs. casual tone.
• Personalization: Using the recipient’s name or tailored content.
• Offers: Testing discounts, free shipping, or other promotional offers.

Here's an Example:

Okay, so we need a very simple example so you know what you're doing. The easiest one to show you is an email subject line. Here's two subject lines I tested for my Newsletter send on 14th October 2024. The email content is exactly the same, just the subject line is different.

Very simply the email went out to 15% of my list, meaning 7.5% got one subject line A (Tesla Triple Threat), and 7.5% got subject line B (Tesla's Bold New Era). Once the results were it, the winning subject line was used when sending to the rest (85%) of my list.

In this case, the "Tesla Triple Threat" subject line won. But...the sample size is way too small, so it isn't statistically significant. So whilst it's an indication I should use the "Tesla Triple Threat" subject line, it's not super reliable that it's better. But hey, it's better than just guessing.

Best Practices for A/B Testing

Here are some key practices for running effective A/B tests:

1. Test One Variable at a Time: The key to successful A/B testing is isolating one variable. If you test too many elements at once, it’s impossible to know what caused the result.
2. Use a Large Sample Size: The bigger your sample size, the more reliable your results. A small group may not provide statistically significant insights. Just be careful doing too big a sample size though, as your only goal is to get a statistically significant result, so you can move on.
3. Run Tests Simultaneously: Timing matters. Running both versions at the same time ensures that external factors (like time of day or seasonality) don’t skew the results. Most email Service Providers (ESPs) for example do this out of the box.

Conclusion: Build, Measure, Learn

A/B testing is an essential part of the Build-Measure-Learn feedback loop in lean startups. By running small experiments, startups can make informed decisions, iterate quickly, and maximize their resources while minimizing risk. If you’re not A/B testing, your money making machine won't get better.