Website Experimentation: A/B Testing
Have you ever run an A/B test on your website? If not, this article walks you through step by step on how to do just that.
While A/B testing has become more and more popular within direct response fundraising, often times website based A/B testing is an area often overlooked within testing, optimization and experimentation. And with the end of Google Optimize, even more organizations are going to miss out from the benefits of website based experimentation as a way to exponentially grow their online revenue.
This article breaks down why you should run A/B testing on your website, how it works, and what to do to get started.
Already know how to run website based A/B tests but just don’t have the time to actually do it?
Generosity X offers full service website based testing services (also known as CRO agency services) with regular bi-monthly reporting starting at $1,295 per month.
Oh, and if we don’t find improvements that justify the costs of our services - we’ll REFUND you the difference.
What is Website based A/B Testing?
You may have never heard of A/B testing before, so let’s start there. A/B testing, also known as split testing, is a method used in marketing to compare two versions of something (like a email, direct mail letter, digital ad or of course, a webpage) to determine which one performs better.
Let’s say you aren’t sure what will drive more donations - including images in a email compared to an email with no images. You could guess what you think would work best or you could split your email list in half and actually see which one delivers more donations!
For the rest of this article, we may refer to “A” as the Control Group and “B” as the Treatment Group. And if you want more information about A/B testing in general, NextAfter has a guide that goes over it in depth that you can find right here.
Website based A/B Testing is a bit different than other types of A/B testing - particularly tests done by direct mail and email. In direct response marketing, most tests are done by splitting a list and sending the piece of marketing material out and measuring the results. With websites, you may have some people come back to the website 10 times within a month, and others who don’t come back at all. How do you “split” your audience when doing A/B tests on a website where the website is a living place people can engage with differently over time periods?
Major platforms, do their best to account for this by assigning different traffic either the A or B version so your tests are measured by Monthly Tracked Users (MTU). So if you come to the website when an A/B test is being run, you’ll be assigned as part of group A or B so that you’ll continue to see that version as you come back to the website (assuming you are using the same device and haven’t cleared your cookies).
This is also part of the reason why most Website A/B testing platforms rely on the Bayesian method of inferential statistics. For the purposes of this article and without blowing up word counts, we won’t be diving into why this is. But if you want to geek out to it, feel free to message me on LinkedIn.
Also, for the sanity of this article, we’ll focus on A/B testing which is one type of Website Experimentation (like how Brie is one type of Cheese) but you may read both throughout this article.
Why Run Website Testing
So we talked about what A/B testing is, but why would anyone actually bother with it? Well the primary reason is oftentimes, we are NOT our donor. And why we think someone gives may be entirely different compared to why they actually give. And when we A/B test, it gives us the advantage.
Conversion Agency is a major CRO agency based in Vancouver that works with some of the biggest brands in the world, and they put it perfectly:
In the gambling industry, the phrase “the house always wins” is a popular adage. There’s a very good reason for this. While the casino doesn’t necessarily profit from every game played, the odds are tilted in its favor so that across a large volume of games played, the cash always ends up back in the coffers. Players may walk out of the casino with more or less money than before, but the average person leaves with less.
Experimentation is not all that different from the casino in this case. The majority of companies may not have a winner for every experiment, but with enough tests, the odds are good that you will come out ahead.
So what does this have to do with experimentation and CRO?
The beauty of experimentation is that the downside is capped (you don’t ship a losing experience), but the upside is infinite (there’s no limit on how much you can lift a conversion rate, even >100%).
With an extensive upside and limited downside, CRO is a great example of a convex system. The “disorder” in this example is every time an A/B test is introduced to the system. Each A/B test (called a trial) creates uncertainty, and thanks to convexity, enough trials lead to positive results over time.
The house always wins.
A/B testing and creating a culture of A/B testing doesn’t mean you’ll necessarily always create a winning test, but by engaging in testing regularly, you’ll overtime find winners that justify the few losers.
Website based A/B testing is also an accessible way to test for more organizations. Compared to email and direct mail tests where you need a significant volume to run a statistically relevant test, website based testing often benefits from the fact that along with high traffic, you can also run tests for longer period of times to make the test more relevant because you catch more traffic to help measure whether your test makes any meaningful difference.
How to actually get started with Website Based A/B testing
Okay, if you’ve made it this far, you likely believe in benefits of website A/B Testing and want me to get to the point already.
In order for the vast majority of organizations to run A/B tests, you’ll need to find a website A/B testing platform that meets your needs. Supposedly, organizations can run A/B tests without using a product, but according to Chat GPT “it can be challenging”.
Generosity X partner with VWO for our A/B testing since it offers a couple of key things that are important for our clients in driving results. I’ll list them below so that you can use this information when deciding on what platform works best you:
Use a platform that offers a visual editor so you don’t require the use of a developer and if you have web development capacity, find a platform that allows for custom javascript to be added to run more complex and meaningful tests. Because we have an in house developer/programmer, this was really important for our team and clients.
Find a platform that fits your price point and if don’t already run A/B testing, start with a platform that’s free or affordable knowing you can always upgrade down the road. VWO has starter plans that are free which means you can jump in and showcase the value of A/B testing to your team without having to invest heavily from the start.
Ensure that whatever platform you use can allow you to ensure mutually exclusivity of your tests. This will make sure that if you run more than a single A/B test, you don’t compromise the results by having additional variables influence the results. If you plan to run multiple A/B tests at the same time, this is a non-negotiable.
Ensure the platform you use allows for testing so that whatever you modify can be double checked. Not only is this important for ensuring that your test is firing events as it should but that your modification didn’t affect something else on your website (particularly your mobile experience).
Use a platform that offers different kinds of testing - some tests are based on small tweaks (50 words vs. 100 words) while other tests are based on trying something totally new and unique. Ensuring your testing platform allows for types of tests - like split URL testing - will help give you flexibility to try different things.
What to actually try testing!
Okay so you want to test, and you have a great platform to actually do website testing with. Now, you’ve gotta figure out what you want to test!
Before you jump in, there are a few things that I suggest to all organizations wanting to run A/B Tests:
Look at your results and analytics to see if you can find any trends that may make you ask questions like “Why is this like that instead of like this?”. Is your median gift amount from online donations higher than your the default amount on your donation form? Do most people click the donation button in the header image instead of on the top navigation? Do users spend time on a particular page compared to others? From there, by asking questions and challenging why things are like that instead of defaulting to “things have always been like that”
Ask people outside of your organization (donors, yes but family & friends are great too) what questions they have on the website as they go through the pages, etc. Do they feel like certain website aspects are unclear, or that certain things matter to them over others? Try to find people who have never donated to your organization to offer their opinion to help challenge assumptions and expand our perspectives. They may be wrong but a test will determine that one way or another.
See what others are testing. NextAfter offers a library of all their tests on their website and other major CRO agencies working in the for-profit space often will post tests in blog posts online. It may not always translate over perfectly, but it’s a great place to draw inspiration from.
Determine what you really want to accomplish. A test to improve email sign ups compared to one that drives more donations or a test that drives higher donations. Yet ask a fundraiser or marketer about any of these and they will tell you that they all matter! It’s up to you to determine what you believe to be the most beneficial for your organization right now, at this time.
Testing is something that more and more organizations can benefit from. And when we all expand our impact by more effectively guiding donors to make a donation, not only does that benefit our world, it also helps more donors become better versions of themselves.
What generosity x clients have to say
"I would give 6 stars if I could. Matt Hussey & Generosity X continue to make a significant impact on our organization. We have benefited greatly from their fundraising knowledge, coaching and individualized support. I would highly recommend Generosity X to any Not for Profit looking to grow their capacity and reach their mission and vision goals!"
- Cathy, Executive Director
"Matt [at Generosity X] is an exceptionally talented digital strategist. I highly recommend him if you're an organization looking to build or grow your digital fundraising portfolio."
- Nam, Director of Development
Want more resources?
Sign up for our email to get great fundraising resources directly to your inbox:
