There is no one right way to customize your ad. Relying on only one setting, text, audience is a failed strategy. For an ad campaign to succeed, you need to experiment. In time immemorial (last year), we all tested various sets of settings manually, but now Yandex.Direct has a tool that allows you to automate the A / B testing process. This magical tool is called Experiments.
When you run a test, the system randomly divides the audience into segments. You determine the number and size of the segments yourself. At the same time, the samples can be different – you can set the size of each group from 1% to 99%, so that the total is 100%. Now “Experiments” are available on request, and they are connected in a couple of days.
It is imperative to experiment at the start of a new campaign. We launched this tool on an old project with a small budget – to try and use it further on other projects. We will tell you what came of it.
Customer: “Zaym Garant” – the company issues microloans to individuals.
Region: The Republic of Khakassia.
Budget: 15,000 rubles (for advertising per month).
Problem: we run ads from 06:00 to 24:00, but in this interval there are time periods that do not bring conversions, spending the budget without return. Before the launch of “Experiments”, we had to poke around in Metric and figure out if there was an effect from night shows. We manually discovered that running ads at night is ineffective, and now the task is to determine the periods in the range from 6 to 24 hours when advertising does not produce the desired results and reallocate the budget.
Decision: plugged in “Experiments” to do A / B testing for the day and evening. The new new Yandex.Direct tool simplifies analysis.
Introductory information about the project
We have been cooperating with the Zaym-Garant company since 2015. Over the years, we have carried out all kinds of optimization. The project is advertised for 150 phrases and receives about 300 clicks from ads per month with an average cost of a click of 52.47 rubles. The semantic core does not require a large budget, because the traffic for keywords in the region is not that large.
Connecting the “Experiments” tool in your account
The tool is relatively new, so it was not connected to our account. To access it, you need to contact a Yandex manager or leave a request to connect the tool through the form in the “Contextual Advertising” section. Now split testing is connected in experimental mode for each client separately; it is not yet possible to connect the tool on the agent account.
In the application, it is necessary to indicate the login, the client’s domain and the approximate plan of the experiment (this is such a formality that you can not write anything especially – we just wrote that we want to test the new tool). Such a request will go to the manager, and he will open access to a new tool. But not at once. You have to wait a couple of days. After connecting, the “Experiments” tab will appear in Yandex.Audience. We will work with her.
Setting up experiments
Before proceeding with the setup, think over a hypothesis that you will test using the split tests. Someone will say that hypotheses are limited only by your imagination, but it is better to start from a real problem. For example, our hypothesis was that the same ad would generate more conversions at a particular time of day than another. To test it and find out when it is better to advertise for a microfinance organization, we decided to compare the effectiveness of advertisements in different time periods:
- from 06:00 to 15:00
- from 15:00 to 24:00
To turn off the period from 00:00 to 06:00, we had to spend time analyzing the data in Metrica. Now, after completing the experiment, all information can be seen in one report.
The first step is to create a new experiment: we come up with a name for it, choose a counter and divide it into segments (two or more).
Important! Experiment Segments and Metric Segments are not the same thing. These are different segments that are not related to each other.
Click “Create Experiment”. There is no need to set up access, because we do not do everything in an agent account. The tool is not yet connected to agency accounts.
The experiment for A / B testing is ready and looks like this:
If you click the “Results” button, Yandex.Metrica will open with a new report in which the segment will be counted for all campaigns. We will conduct testing within one search campaign, and it will be more convenient to view the result in the Yandex.Direct Report Wizard.
Preparing a campaign for an experiment and launching
For the experiment, we took an advertising campaign on a search. Before connecting the configured segments, you need to make two copies of the original campaign (by the number of test segments).
The screenshot shows the original campaign and two copies of it with notes – in our case, the segment names have been added.
Next, you need to set all the necessary settings for campaigns based on the conditions we have defined for the segments. That is, we set up the required time targeting in each campaign: for segment A, we set the time from 06:00 to 15:00.
Important! Timed targeting cannot cover less than 40 hours on business days.
When the targetings are set, scroll down the settings until we see the “Experiments” section. In it, select the desired experiment and segment:
Leave the rest of the settings unchanged. Remember to save.
For another campaign, we repeat the same sequence of actions, but we already use segment B with a different time targeting.
Before launching new campaigns, you need to set the end date in the settings of the original AC – the day on which you plan to launch two new campaigns. Well, for new campaigns, you need to set a launch date. For the purity of the experiment, it is important that new campaigns are launched on the same day. In the screenshot above, you can see that the original campaign ended on June 24, 2019, and two copies of it with experimental segments were launched on the same day.
Analyzing Results in the Report Wizard
We ran ads for three weeks and then decided to see the results. During this time, statistics should be accumulated sufficient to draw conclusions about the effectiveness of the advertising campaign.
To see the results, select the campaigns in which the A / B testing was performed and look. You need to build a report in the “Report Wizard” tab. To do this, specify the following values:
- Grouping: for the selected period.
- Target data: sending any order form. Here we select the main goals for which we will draw conclusions. In our case, a universal purpose is indicated, which collects data about the submission of all forms on the site.
- Attribution: first transition. For advertising, it is better to use this attribution: here the first transition of a user to the site in the last 180 days is considered. This way you can track the source of the ad that brought the customer and influenced all subsequent interactions with the site.
- Report period: any. We have chosen a three-week period from the launch of the campaign.
Then we mark the parameters to be compared.
Below we mark the filtering condition “Experiments” and select the desired experiment (if you have several of them).
The campaign with segment A (air time – from 06:00 to 15:00) went into a serious gap:
- the number of conversions is significantly higher – 28.57% compared to 13.04% during the display period from 15:00 to 00:00;
- the average cost per click is almost 20 rubles lower;
- the average position is higher – 2.01 versus 2.48;
- refusals are four times less – only 5.26%, while in the period from 15:00 to 00:00 there were 21.43% refusals;
- the cost of achieving the goal is about 1.5 times lower.
At the same time, the rates in both campaigns are the same and they were launched at the same time.
Experiments from Yandex is an easy-to-learn tool that allows you to generate reports that are easy to analyze. Earlier, when conducting split tests, to find out the results, you had to shovel the entire Metric, but now you can generate one report, and it will reflect all the important data on the required campaigns.
Conveniently, Experiments are universal. You can build any hypothesis and test it on A / B tests. It is convenient to compare display campaigns, different media plans, campaigns in YAN and on search, automatic strategies and manual ones (if you need to test three strategies, you will need to set up three segments), and so on.
We worked with search campaigns and tested their performance based on different time targeting. For more than three weeks, we have accumulated enough statistics to understand in which segment the indicators are more interesting, and redirect budgets to a more effective campaign. Now, having tested Experiments on a small project, we are launching the tool on large clients with a budget of 100,000 rubles or more.