Get to know about the steps required to launch an A/B test and reuse it
If you want to start running mobile A/B tests in your app, getting your first experiment out the door can be a fast process.
All you need to do is install a mobile A/B testing tool, come up with a test idea, create your variations (either with help from engineering or with an A/B testing tool with a Visual Editor), and hit ‘Launch’!
While running a single test doesn’t have to be laborsome, getting buy-in for mobile A/B testing and delivering value through continuous experimentation takes more time. You’ve got to get resources, engineering help, executive sponsorship, and more.
In this blog, we’ll outline the stages of implementing and maintaining an effective app A/B testing program.
Like any important initiative at your company, it’s vital you get a buy-in for your A/B testing program. That means securing a sponsor (hopefully an executive that manages resources) and investment in the A/B testing tools and people you’ll need to execute effectively.
The key here is pinpointing the pain points you hope A/B testing will help you solve, and making a business case for implementing an experimentation program by tying it to larger company goals.
Possible use cases for mobile A/B testing could be:
Tip: Pull together case studies, stats, and infographics that show the benefits of mobile A/B testing to pique the interest of your team.
Once you’ve secured resources and set some initial goals, you’ll need to create a consistent process to help you and launch your first few tests and execute effectively in the long-run.
Here are the overarching elements you’ll need to establish a robust internal testing process:
You’ll need to decide how you’ll conduct your A/B tests, whether that’s through buying a mobile A/B testing platform or building an in-house solution. (This blog covers how to decide between the two.)
At this stage, it’s also important to figure out how much time engineering can invest in supporting A/B testing and how you’ll track and analyze results—which may inform your decisions around building a solution from scratch or moving quickly with a vendor.
You’ll want to use a roadmap or tracking tool to manage your experiment queue. Having a schedule will help your teams stay aligned and offer visibility into your progress. By creating a calendar and consistent pattern of testing, you’ll also keep experimentation top of mind with your teammates.
You’ll want to meet with cross-functional teams to brainstorm test ideas. (Keep the conversation focused by reiterating the goals you established in Stage 1.)
Prioritize the top A/B tests you want to launch in your app first, and then record the rest to create a backlog of ideas you can tap into later (either in your roadmap or somewhere else). Regularly schedule these brainstorming sessions so you can have testing initiatives consistently on-the-go.
These meetups will also help you stay up to date on team-based or company-wide objectives that can be supported through mobile A/B testing.
Once you’ve run and launched your first test (which we’ll cover in the next section), you’ll need to share results with cross-functional teams.
Decide whether this will go through an in-person standup, email, sharing a report, or a combination of methods. Note that some of the tests may also be inconclusive.
You’ll need to adjust and re-run inconclusive tests to gain insights or design a different test—a process which will likely reveal crucial learnings in and of itself.
Tip: Set realistic expectations for your first test. Don’t plan to impact your top metrics right away.
Take an iterative approach to testing instead by testing smaller changes (like tweaking parts of your mobile onboarding messaging) that may lead to the larger outcomes you desire (like increasing sign-ups or activation).
Making a massive app change all at once can negatively impact the user experience in ways you likely can’t predict early on in your testing process.
Now that you understand the elements needed to set up your larger A/B testing initiative internally, here are the steps to launching individual A/B tests that you can re-use over and over again:
At this point, it may be valuable to establish goals for the frequency and velocity with which you run experiments.
Before selecting these goals, you may want to wait until you’ve executed a few tests first to get a sense for your potential speed—and any roadblocks that may prevent you from achieving your goals.
Tip: Our team recommends aiming to launch a few A/B tests in your first month of testing (1-3); then a few more by Day 60 (4-6); and aiming for 10+ tests by the 90-day mark.
Now that you have some initial results, it’s time to take stock of your progress to see where you can improve your processes or where you may need to adjust your goals.
It’s also vital to set up a larger, recurring review of test insights with key stakeholders, sponsors, and executives from relevant departments. (We recommend it at least a quarter.)
In these meetings or presentations, dive into the overarching learnings you’ve gleaned from your tests around:
These meetups also present an opportunity to see if company-level goals or objectives have changed, and to think through how that can be incorporated into your testing roadmap.
Many product management and engineering teams get excited about mobile A/B testing—only to stop testing once they run into a few roadblocks, or have their focus pulled away by other development priorities.
Establishing your top goals and metrics, getting a high-level buy-in and setting aside resources for mobile A/B testing before you begin can help prevent this organizational drop-off.
So will actively engage as many team members as you can in your experimentation process. You’ll also need to continuously assess and refine your process so that it fits into the development process and supports ever-evolving product goals.
To learn more about mobile experimentation, check out Taplytics’ Ultimate Mobile A/B Testing Guide (which includes a list of our 53 favorite resources for mobile app makers!).
Also, if you are interested in reading more about the trending topics like mobile app testing straight from the mobile app industry, then make sure to follow MobileAppDaily by clicking on the ‘Subscribe’ button now.
She is an experienced business and marketing consultant of the mobile app industry; advising on anything from the perfect branding to the latest tech releases. She is passionate about writing well-researched reports to help the app owners and the mobile app industry audience. She has a vibrant touch that goes well in her writing as well.