A/B Test

1. Functional Overview

A/B Test is a comparative test that divides users into a control group and a test group in proportion and uses different configurations for different groups. You can use different advertising strategies in different test groups to compare the results to help you verify which advertising strategy is more effective.

TopOn A/B Test supports the following functions:

(1) Support multiple test groups, TopOn A/B Test can set up to 10 test groups

(3) For A/B testing of ad positions, multiple TopOn ad positions can be selected simultaneously. For traffic segmentation A/B testing, only one traffic segment can be chosen for A/B testing at a time. Meanwhile, TopOn supports viewing A/B testing data from the application dimension.

(4) After the A/B Test is over, you can select the best test group with one click

2. Operation Guide

2.1 A/B Test usage process

2.1.1 Create A/B Test

(1) You can create and manage A/B Tests through Advanced → A/B Test

(2) When creating an A/B Test, you can set multiple A/B groups (up to 10 groups), and you can select multiple TopOn placements for A/B Test in the same app(The traffic segmentation A/B testing only supports the creation of a single traffic segmentation.) Note: For the same ad placement, only one type of A/B testing can be created.

(3) After the A/B Test is created, the original WaterFall configuration of the mediation will be automatically synchronized to the group with the smallest ID in the A/B Test (for example, if three groups A\B\C are configured, it will be automatically copied to group A). How to synchronize the original WaterFall configuration of the mediation of other groups (B/C)? You can use the Duplicate WaterFall function

(4) The effective time of A/B Test can be selected as Immediately or Assign

(5) You can use the A/B Test status filter or search function to quickly find the A/B Test experiments that have been created in the current app

2.1.2 Set WaterFall for A/B Test groups

(1) After you create an A/B Test, you can configure Segment and Ad Sources for each A/B Test group in Mediation.

(2) You can use the Copy Segment and Copy Ad Source functions to quickly complete WaterFall configuration

(3) If there is no available ad source in the test group, TopOn will "Automatically allocate impression ads" by default, that is, use the ad source of other test groups for impression. You can also choose ”Do not impression ads in this group“

2.1.3 View A/B Test data in mediation

(1) The group data of the current A/B Test will be impressioned in the corresponding group. You can adjust the WaterFall configuration in the current A/B Test group

(2) The data generated by the old cache strategy of TopOn SDK (not the strategy of the current A/B Test) will be impressioned on ”Historical traffic data“

Note: Historical traffic data only supports viewing data, and WaterFall configuration cannot be modified

(3) In Historical Traffic Data, you can filter all the historical A/B Test data you want to view by the A/B Test group name

Note: In the historical segmentation data, only the segmentation of A/B testing and the status of ad sources in the enabled state before the end of the test will be displayed. Ad sources that were turned off during the A/B testing will not be recorded. For the complete data, you need to check the full reports.

Note: After distinguishing A/B groups, please pay attention to metrics such as "estimated data" when observing data. For example: estimated eCPM, estimated revenue, and estimated ARPU.

The metrics with "API" in the A/B group are data of the entire code bit dimension, that is, the actual revenue of this code bit. In fact, no split will occur.

For example: CSJ code position "xxxx" is configured in an A/B Test with a 60/40 distribution ratio, and the actual revenue API is 100 RMB. The estimated revenue in group A is 60 RMB, and in group B it is 40 RMB. However, the revenue API in both groups will be consistent with the "actual revenue pulled back from the mediation network", that is, whether watching in group A or group B, the revenue API of CSJ code position "xxxx" will be equal to 100.

2.1.4 View A/B Test data in A/B Test

(1) You can view A/B Test data by app dimension on the A/B Test page, and you can also view test data by placement dimension

(2) A/B Test supports data metrics:

• DAU
• DEU
• Estimated revenue/DAU
• Estimated revenue/DEU
• Estimated revenue
• Estimated eCPM
• Requests
• Request fill rate
• Impressions (from March 7, 2023 Starting from 0:00 (UTC+8), the statistical caliber is adjusted from "the number of times TopOn calls the advertising platform SDK to start impression" to "the number of callbacks TopOn receives from the advertising platform SDK for successful impression")
• Impression rate
• Engagement rate
• Imp./DAU
• Imp./DEU
• Clicks
• CTR
• isReady
• isReady rate
• Impressions (old) (original impression statistical caliber data, the number of times TopOn calls the advertising platform SDK to start impression)
• Cut-in number (DAU data of the original placement dimension, the number of independent users who have requested any placement in this A/B Test every day, you can view the actual AB test cut-in ratio)

2.1.5 View data trends in A/B Test

(1) You can view the changes in A/B Test within the selected date range through the Data Trend chart of A/B Test

(2) Click Date Detail on the right side of App Data or Placement Data to enter

(3) In the data trend chart, you can select the corresponding data metric to view the trend chart, or you can export Date Detail

report  for each day

2.2 End A/B Test

(1) When ending A/B Test, you can specify all placements and select the best grouping strategy with one click

(2) You can also choose one or more placement(s) and select the best grouping strategy separately

2.3 View Historical A/B Test Data

(1) After the A/B Test is completed, TopOn SDK strategy cache problem. Unselected grouping strategies will generate legacy data within a certain period of time. You can view it in the following ways:

After the AB test is over, if you find that the placement data queried in the full report is inconsistent with the data impressioned in the mediation, you can select AB test in the data dimension of the full report to further determine whether the data difference is caused by the old AB test strategy

(2) View the finished AB test data in A/B Test

① After the A/B Test is finished, only the data of the optimal AB test group you have selected will be impressioned on the mediation page. The data of other AB test groups that have not been selected will not be impressioned in the mediation page. If you need to view the data of the unselected AB test group, you can view it in Historical Traffic Data, or go to full report to view it (select AB Test in the data dimension)

② After the A/B Test is over, if you need to view the historical data before the AB test was started on the mediation page, you can view it in Historical Traffic Data, or go to full report to view it (select AB Test in the data dimension)

3. FAQ

(1) How to allocate devices for the A/B Test function?

If the AB strategy remains unchanged, the device ID will be fixedly allocated to one of the groups. If the strategy changes, such as modifying the volume cut permission, the device ID will be reallocated.

(2) The cut-in ratio set in the A/B Test does not match the cut-in ratio of the actual placement, but the DAU cut-in ratio of the App is close to the cut-in ratio set in the A/B Test?

Under normal circumstances, it is normal for the gap between the cut-in ratio set in the AB experiment and the cut-in ratio of the placements in the experiment to be 5%.

When the gap exceeds 5%, you can check whether there are any test groups in all the placements participating in the experiment that have no ad source configured. When the test group is not configured with any available ad source, TopOn adopts the "Automatically allocate impression ads" strategy by default, that is, the traffic of this test group will be cut to other test groups with available ad sources, thereby affecting the actual cut-in ratio of the placement.

(3) When a single placement participates in an A/B Test, the A/B Test results of the placement are inconsistent with the A/B Test results of the App dimension?

Under normal circumstances, the A/B Test results of a single placement will be consistent with the A/B Test results of the App dimension. When there is a discrepancy, it may be that the samples participating in the A/B Test are unevenly cut.

It is recommended that you close the existing A/B Test and create a new A/B Test. In the new A/B Test, you can consider using the AABB test (two AA groups as the control group and two BB groups as the test group). When the test results of the two AA groups are close, it means that the A/B Test samples are evenly cut. This A/B Test can consider the comparison effect of the AB groups.

(4) When multiple placements participate in the experiment, the summary data of each placement participating in the experiment is inconsistent with the experimental results of the App dimension?

When this happens, it means that the test results of your current placements participating in the A/B Test and the overall App dimension show the "Simpson's Paradox". In this case, you can select the optimal strategy test grouping strategy based on the A/B Test results of each placement.

At the same time, it is recommended that you create different A/B Tests for the above placements participating in the A/B Test and re-verify the existing test conclusions.