A/B tests

A test compares two versions of one offer. Version A is the offer as it stands. Version B can offer different products, a different discount or different wording. Either version can sell the best pairing for each order instead of products you pick. When both versions sell the best pairing, they can also differ in how many cards they show. One customer in two sees version B on the post-purchase page, decided by the order so nobody sees both there, and everything else stays the same for both versions: who sees the offer, and when.

Start a test

1. Open the offer from Offers and go to What it offers. The offer has to be saved and turned on first.
2. Under the products, click Test a second version.
3. Version B starts as a copy of version A. Under What version B sells, select Products you pick or The best pairing for each order. Then change its products, if you pick them, its discount or its wording. When it sells the best pairing, Products on the page sets how many cards it shows. If the offer has counter-offers and shows its products one after another, a customer who declines the first card sees a counter-offer in place of the next card when one can be offered on that order. The cards after the first are then shown only to customers who said yes to it, and only when Keep offering after they accept is turned on.
4. Under How long it runs, select the smallest difference you want the test to be able to find. A smaller one needs more views, and the page says roughly how long that takes at this offer’s traffic.
5. Click Start the test.

A view is a post-purchase page that showed one of the two versions. A page that shows several products counts as accepted once, when the customer accepts at least one of them, and the money of every product accepted is added. Your own test orders count in neither version, checked since the test started (over the last 60 days on a store that has not yet approved access to all its orders). When an offer was shown on too few post-purchase pages in the last 30 days, or nobody accepted it there, the test starts from your store’s rate, or from a typical one, and says which and why. The Thank you and Order status pages show version A while a test runs, so an offer has to be on the post-purchase page to be tested.

While it runs

• The offer is locked, so the two versions stay comparable. To edit it, end the test by keeping one version.
• An offer cannot be paused while it is being tested. The test itself can.
• Pause stops the test: every customer sees version A until you click Resume, and nothing is counted meanwhile.
• Each version needs a set number of views, and the test card shows how far each has got. A lead before both versions get there is usually luck, so no winner is named until then.
• A test also runs for at least a week, so that every day of the week is in it. Customers on a Saturday do not buy like customers on a Tuesday.

The result

Once both versions have their views and the week is up, the test says one of two things: one version won, on the money it added per page shown, with the share of customers who accepted each beside it, or there is no real difference and you can keep whichever suits you for another reason, such as margin or stock. It tells you when version B does worse, not only when it does better.

Keep version A ends the test and version B goes. Keep version B makes version B’s products, discount and wording the offer’s own, and version A goes. When version B sells the best pairing for each order, keeping it makes the offer sell the best pairing the way version B does, with its number of cards, its discount and its wording, and the offer’s counter-offers go. You can end a test either way at any time, before the result is in, unless every product of version B has since been deleted from your store. A test cannot be picked up again where it stopped.

Some offers cannot be tested usefully: one that nearly every customer already takes, or one so few take that no test could tell two versions apart. The test page says so rather than starting a test that would not finish.