Smarter A/B testing: The latest enhancements from GameAnalytics

Launching a game is just the beginning. To keep players engaged and attract new ones, you need to operate your game as a live service – continuously rolling out new features, updates, promotions, and in-game events to maintain and maximize engagement and monetization.

That’s where data-driven experimentation comes in. A/B testing ensures that every change you make has a measurable impact, rather than relying on guesswork (and hoping for the best ?). To support your LiveOps strategy, we’ve been co-creating with Supercent, and built powerful new improvements for our A/B Testing capabilities – giving you greater flexibility during test creation and deeper insights into your results.

What’s changed?

• Creating your A/B tests
  • Automatically include new builds
  • Set your sample size
  • Define the enrollment rate
  • Flexible variants and variant distributions
  • Set an end date as your completion criteria
  • Run up to 100 tests at a time
• Analyzing the results
  • Revamped results table
  • 15+ new metrics
  • Dive deeper in Explore & Funnels
  • Time series & statistical analysis
  • Email notification

Creating your A/B tests

Automatically include new builds

You can now include builds not yet tracked in your a/b tests. Once selected, players from the selected build(s) and beyond will be included in the test.

Set your sample size

You can now control how many users to enrol in your a/b test. You’ll also be guided to include enough users to create a statistically significant result.

Define the enrollment rate

Set the probability of a user being enrolled into your test – e.g., a 10% value means each player has a 10% chance of being enrolled.

Flexible variants and variant distributions

Add up to 10 variants at the same time to test your hypothesis, and customize the variant sizes – e.g., assign 50% of your players to one variant and divide the rest as you like. The choice is yours.

Set an end date as your completion criteria

You can now run an a/b test for a few days, a week, or even longer. Results will be ready whichever date you choose.

Run up to 100 tests at a time

Lots of hypotheses? No problem. Test them all with the ability to run up to 100 a/b tests at the same time.

Analyzing the results

Revamped results table

We’ve made it easier to understand your a/b test results with a revamped results table. You can also create a custom results view, so you can directly compare the metrics that matter most to you.

15+ new metrics

Get a comprehensive understanding of your a/b test by analyzing even more metrics:

• Retention (Day 1 / Day 2 / Day 3 / Day 4 / Day 5 / Day 6 / Day 7 / Day 14 / Day 28)
  • The percentage of users who return to the app on the specific day after their first session.
• Playtime per user (Day 0 / Day 1 / Day 2 / Day 3 / Day 4 / Day 5)
  • The playtime per user spent on the specific day after install.
• ILRD count per DAU (Interstitial, Day 0)
  • The number of interstitial ad impressions per daily active user on Day 0 (first day of use).
• ILRD count per DAU (Rewarded, Day 0)
  • The number of rewarded ad impressions per daily active user on Day 0.
• Ads shown per user (Rewarded, Day 0 / Day 3 / Day 7)
  • mean_count_per_user of rewarded video ads watched on the specific day.
• Ads shown per user (Interstitial, Day 0 / Day 3 / Day 7)
  • mean_count_per_user of interstitial ads watched on the specific day.
• Total ads shown (Interstitial and Rewarded, Day 0 / Day 3 / Day 7)
  • The total number of interstitial and rewarded video ads watched on the specific day per user.
• ARPDAU (ILRD)
  • The average revenue generated from in app ads per daily active user.
• ARPDAU (IAP) and (IAP, Valid)
  • The average revenue generated from in-app purchases per daily active user.
• ARPAU (ILRD)
  • The average revenue generated from in app ads per active user.
• ARPAU (IAP) and (IAP, Valid)
  • The average revenue generated from in-app purchases per active user.
• PUR (Day 0 / Day 7)
  • The percentage of users who make a purchase until the specific day (Paying User Rate).
• Cumulative Revenue (IAP, Day 0 / Day 3 / Day 7)
  • Total in-app purchase revenue per install, generated in the first X days after install.
• Cumulative Revenue (ILRD, Day 0 / Day 3 / Day 7)
  • Total impression purchase revenue per install, generated in the first X days after install.
• Cumulative Revenue (Total, Day 0 / Day 3 / Day 7)
  • Total in-app purchase and impression revenue per install, generated in the first X days after install.

Dive deeper in Explore & Funnels

You can now Group or Split data in Explore by a/b test or a/b test variant, giving you greater flexibility and deeper analysis capabilities outside of the a/b test results page.

ℹ️ Our Explore tool allows you to visualize a multitide of prepared metrics, or create your own using filters, aggregations and granularity options.

You can also filter by a/b test in Explore and Funnels.

Tip?: Once an a/b test is complete, hover over a metric and click the ‘View in Explore’ icon. This will open Explore, with your goal metric selected, a/b test and variant filters applied with the corresponding start and end date. Happy exploring!

Time series and statistical analysis

Use the time series to analyze any metric available on the results page.

You can also use the statistical analysis to identify a winning variant. Note: This is only applicable if a goal metric is selected as the completion criteria (as opposed to end date).

Email notification

You can now get an update in your inbox once your a/b test is complete.

What’s next?

And that’s not all. We’re working on enabling targeting All and/or Returning Users, updates to our Remote Configs and more. If you have any suggestions, don’t hesitate to share your thoughts with us through the feedback portal.

To stay up to date on all things product, subscribe to our newsletter.