Uplift icon

Adyen Uplift experiments

Learn more about how you can start, evaluate, and stop experiments.

You can run different types of experiments to validate and evaluate the impact of changes you make to settings on your account. For example, you can run experiments to see if a different setup is likely to lead to an uplift in conversion or revenue, or compare the performance of two risk profiles.

Experiments are not simulations. They are performed on real transactions, processed with your live Adyen account.

Requirements

Before you begin, take into account the following requirements and limitations.

Requirement Description
Integration type Make sure that you have built an online payments integration that supports Adyen Uplift. For more information, see the detailed requirements.
Customer Area roles All Customer Area users can see the Experiments page in the Revenue & risk section.

To start a checkout experiment, you need one of the following roles:
  • Merchant admin
  • Merchant manage Uplift optimize settings combined with Change payment methods
To start an optimization experiment following a recommendation, you need the following role:
  • Merchant manage Uplift optimize settings
To start a risk experiment, you need the following role:
  • Risk admin
Limitations Optimization experiments:
  • You can only start optimization experiments following a recommendation.
Checkout payment method experiments:
  • You can only start experiments on payment methods that you have already activated, and you must have at least three active payment methods.
  • You cannot start an experiment on primary payment methods such as credit or debit cards.
Checkout dynamic payment method ordering experiments:

Experiment types

There are different types of experiments you can run to validate the effect of making changes to settings on your account:

Checkout experiments

Run a payment method experiment to analyze the impact on conversion and cost of temporarily removing a payment method from your checkout page in a country or region. You can do this, for example, to learn if removing a costly payment method negatively impacts conversion, or not.

Or, run a dynamic ordering experiment to see the effect of dynamically ordering payment methods on a cost or conversion optimization strategy.

Optimization experiments

An Adyen Uplift recommendation can include the option to start an experiment when changing one or more settings is likely to lead to an uplift in performance or revenue.

When a recommendation gives you that option, you can start an optimization experiment on a portion of your transactions. You get recommendations and can start experiments that apply to your integration, and that are relevant for the countries or regions you operate in.

By analyzing the results during the experiment, or after you stop it, you can validate the effect of any changes. You can then choose to change to the recommended settings, or decide not to act on the recommendation.

Risk experiments

You can run a risk experiment to compare the performance of two risk profiles and determine which performs better. For example, you can run a risk experiment to see if changing, adding, or disabling a risk rule has the effect you want on your fraud rate.

See run an A/B test experiments for details on how to start, evaluate and stop risk experiments. Risk experiments require Protect premium.

Start an experiment

Start a checkout payment method experiment

To start an experiment, in your Customer Area:

  1. Go to Settings > Checkout.

  2. Select the Country for which you want to start an experiment.

  3. Find the payment method that you want to exclude from checkout for the experiment, and select the test button.

  4. Select Continue.

  5. For each experiment, do the following:

    1. Enter an experiment Name.
    2. Select the Merchant account from which you want to include transactions for the experiment. If you are logged in on the company account, select a single merchant account from the dropdown menu.
    3. Confirm the payment method and the country or region.

    4. Set a traffic percentage. This is the proportion of transactions that you want to include in your experiment.

      You can change the traffic percentage at any time during the experiment. For example, you can start with a low percentage and gradually increase the percentage of traffic if you see good results.

  6. Select Create to start running your experiment.

Your experiment will continue running until you stop the experiment.

Start a checkout dynamic ordering experiment

To start an experiment, dynamic ordering must be enabled.

In your Customer Area:

  1. Go to Settings > Checkout.
  2. Select the Country for which you want to start an experiment.
  3. Select Experiment.
  4. Select Create to start running your experiment.

Your experiment will continue running until you stop the experiment.

Start an optimization experiment

To start an experiment, in your Customer Area:

  1. Go to Revenue & risk > Uplift overview > Recommendations.

  2. Select a recommendation and then select Create experiment.
    This button is only available when the recommendation allows you to experiment, and when you meet the requirements to run the experiment. For example, certain recommendations require that you process payments in a specific country or region, or that you have enabled a payment method or feature. The recommendation contains details about what is required before you can create an experiment.

  3. For each experiment, do the following:

    1. Enter an experiment Name.
    2. Select the Merchant account from which you want to include transactions for the experiment. If you are logged in on the company account, you can select multiple merchant accounts.

    3. Set a traffic percentage. This is the proportion of transactions that you want to include in your experiment.

      Transactions in your experiment will be evenly split between your current (Variant A) and recommended (Variant B) setting. For example, if you set a Traffic percentage of 20%, the current setting is applied to 10% of your transactions, and the recommended setting is applied to 10%.

  4. Select Create to start running your experiment.

While the experiment is running, the user who created the experiment can edit the name of the experiment and the traffic percentage.

Your experiment will continue running until you stop the experiment.

View experiment results

You can view data from running and completed experiments. At the start of the experiment, it takes about four hours before you can see the first results. After that, the results of your running experiment are updated regularly.

To view data from an experiment in your Customer Area:

  1. Go to Revenue & risk > Experiments
  2. Select Risk, Optimizations, or Checkout.
  3. Select Running or Completed and then select an experiment to view the results.

For each experiment, you can view and compare the performance of the different setting or comparison variants. The data that you can compare depends on the experiment.

Stop an experiment

When you view a running or completed experiment, the results show if the experiment has reached statistical significance. We recommend letting an experiment run until that point is reached. You can then stop the experiment and, if applicable, implement any changes to achieve the variant with the best results.

If you stop an experiment before a statistically significant result has been reached, we recommend to rerun the experiment for a longer time period, or on a larger portion of your transactions.

To stop an experiment, in your Customer Area:

  1. Go to Revenue & risk > Experiments.
  2. In the Optimizations or Checkout tab, go to your experiment on the list, and select End.
    • For optimization experiments, select End to stop the experiment and keep your current setting, or select End & change setting to act on the recommendation and change your settings.
    • For checkout experiments, select End to stop the experiment.

See also