Skip to main content

A/B Testing | Smart Cart

Split test one Smart Cart against another or against no cart at all with Rebuy's built-in A/B testing

Written by Tom

Use this article to split test one Smart Cart against another, or against no Smart Cart at all, and to end the test with a winner live. A/B testing is in your Rebuy admin under Flows & A/B Testing, then A/B Testing, on a page headed Experiments.

When you end an experiment yourself, you choose Set Winner Live to turn on the winning Smart Cart, or Do Nothing to turn one on yourself later. When an experiment reaches the End Date you scheduled, Rebuy turns on a Smart Cart for you, either a clear Conversion Rate winner or the Smart Cart that was live on your store when the test started.

Creating the carts you are going to test is covered in Smart Cart Settings, and turning a cart on outside a test is covered in Smart Cart: Set Up and Publish Your Cart.

Legacy carts are not supported in this feature.

A/B Testing is only available for the latest Smart Cart version. If you are using a Legacy Smart Cart, you’ll need to create a new Smart Cart to utilize A/B testing

What you need before you create a Smart Cart A/B test

A Smart Cart A/B test needs the following in place before it will start, and getting them ready first avoids a half-configured experiment sitting in your admin.

  • The Flows & A/B Testing package on your Rebuy account. Without it the A/B Testing entry does not appear under Flows & A/B Testing in the sidebar.

  • More than one Smart Cart to compare. The usual approach is to duplicate the cart that is live now and change one thing on the copy, so the result tells you what that one change did.

  • The latest Smart Cart version. Rebuy's A/B testing is built for Smart Cart V2, and a Legacy Smart Cart cannot be enabled at all while a Smart Cart test is running. Trying returns Cannot enable a legacy Smart Cart while an A/B Test for Smart Carts is active.

You do not need to enable any theme app embed for a Smart Cart test. The rebuy-ab-testing-embed app embed carries head-injected CSS and JavaScript that only Smart Search experiments produce, so enabling it changes nothing for a Smart Cart test.


Create a Smart Cart A/B test

Creating a Smart Cart A/B test takes you from the Experiments page through the cart selection, an optional audience, the goal, and an optional schedule, in the order of the steps below.

Step 1: Open A/B Testing and create a new experiment

To start a Smart Cart A/B test, go to Flows & A/B Testing, then A/B Testing, in your Rebuy admin, and click Create New Experiment on the Experiments page. The A/B Testing entry appears in the sidebar only when your Rebuy account has the Flows & A/B Testing package.

Smart Cart A/B testing: the Experiments page opened from Flows and A/B Testing then A/B Testing in the Rebuy admin sidebar, with the Create New Experiment button highlighted

Step 2: Name the test and choose the Smart Cart experiment type

The Create New A/B Test window is where you name a Smart Cart A/B test and pick its type. Type a Name that tells you what the test compares, choose the Smart Cart tile under Select Experiment Type, and click Add Experiment. The experiment editor opens with the Smart Cart sections ready to fill in.

Smart Cart A/B testing: the Create New A/B Test window with a test name entered and the Smart Cart tile selected under Select Experiment Type

Step 3: Choose your control cart, variations, and set the traffic split

The Select Control and Variations section is where you choose which Smart Carts a Smart Cart A/B test compares. Pick the Smart Cart you want as the control, which is your baseline, then add the Smart Carts to test against it. The admin's own instruction is Select your control cart and up to 2 cart variations to test in your experiment. To test a Smart Cart against your theme's native cart, choose Do Not Show Anything for one variation.

Set the traffic split. The traffic percentage beside each cart in a Smart Cart A/B test sets how many shoppers Rebuy assigns to that cart. The percentages have to add up to 100, and none of them can be 0. If they do not, Rebuy does not start the test and shows Total traffic does not add up to 100%. or Looks like you have 0% traffic for one or more of your experiment variations.

Each variation can be previewed on your live storefront from this section without changing the test's traffic or results, as covered in How to Preview an A/B Test Experiment. A cart in the test cannot be edited once the test starts, so finish setting up every cart before you continue.

Smart Cart A/B testing: the Select Control and Variations section with a control Smart Cart and a variation Smart Cart selected and a preview link beside each

Step 4: Choose an audience (optional)

The Audience section, below Select Control and Variations, limits a Smart Cart A/B test to one group of shoppers. It defaults to Everyone (All shoppers will be entered into the experiment). To test a single group instead, choose Device, Visitor type, Location, Customer tag, or Klaviyo segment and set its value. Shoppers outside the audience are not counted in the results and see the Smart Cart that was live on your store when the test started.

The audience locks once the test starts. What each option targets is covered in How to Target an A/B Test to a Specific Audience.

Smart Cart A/B testing: the Audience dropdown in the experiment editor open to show the Everyone, Device, Visitor type, Location, Customer tag, and Klaviyo segment options

Step 5: Choose the Experiment Goal

The Experiment Goal section sets the metric a Smart Cart A/B test uses to pick a winner. Choose Revenue to find the cart that brings in the most revenue, or Conversion Rate (cr) to find the cart with the highest share of visitors who place an order. Only a Conversion Rate (cr) test can have its winner set live automatically when it reaches a scheduled End Date.

Smart Cart A/B testing: the Experiment Goal section with the Revenue and Conversion Rate (cr) options

Step 6: Schedule a start and End Date (optional)

The Test Duration section lets a Smart Cart A/B test start and end on its own. Set a Start Date and start time to have the test begin later, and an End Date and end time to have it end automatically. Times run in 30-minute steps in your store's time zone, a test cannot be scheduled to start more than 30 days ahead, and it cannot end more than 30 days after its start. Leave the section empty to start and end the test yourself.

A test that reaches its End Date ends on its own and turns on either a clear Conversion Rate winner or the Smart Cart that was live on your store when the test started.

Smart Cart A/B testing: the Test Duration section with a start date and time and an End Date and time set for the experiment

Step 7: Start the experiment

To start a Smart Cart A/B test, click Start Experiment at the top of the experiment editor, then click Start Experiment again in the confirmation window. A running test appears on the Live tab of the Experiments page. An experiment with only a control and no variation does not start, and Rebuy says Cannot start experiment with only one variant. Please add at least one test variant.

Once the test is running, each shopper is assigned to one variation and stays on it, so a returning shopper sees the same cart rather than a different one on each visit.

Note: In a Smart Cart A/B test, Rebuy keeps each shopper on their variation with a cookie saved in their browser. A shopper can be assigned a variation again, which may be a different one, and can be counted again in the Visitors column when they clear their cookies or browsing data, browse in a private or incognito window, switch to a different browser or device, or come back after the cookie expires. This includes a logged-in customer on a new device, since the variation is saved in the browser rather than on the customer's account.


Test your Smart Cart against no cart at all

To test your Smart Cart against no Smart Cart at all, choose Do Not Show Anything for one of the variations in a Smart Cart test, which leaves the shoppers assigned to it with no Smart Cart. Use it when the question is whether the Smart Cart beats your theme's own cart rather than which Smart Cart is better. Like every Smart Cart test, this needs the Flows & A/B Testing package on your Rebuy account.

Choose Do Not Show Anything for one variation and a Smart Cart for the other when you set up Select Control and Variations. Shoppers in that variation get no Smart Cart, so the cart icon and add to cart open your Shopify theme's native cart instead, as its drawer or its /cart page.

Smart Cart A/B testing against no cart: the Select Control and Variations section with a Smart Cart as the control at 50 percent and Do Not Show Anything chosen for variation 1 at 50 percent

Do Not Show Anything shoppers need a working Shopify theme cart. The Smart Cart and its Custom CSS never load for shoppers in this variation, so they use your Shopify theme's native cart setup instead. Before you start the test, check that any CSS hiding your Shopify theme's native cart lives only in the Smart Cart's Custom CSS box, as described in Smart Cart: Set Up and Publish Your Cart. If any CSS hiding your Shopify theme's native cart is in your theme's files, or your theme's cart drawer has been removed, shoppers in this variation will have no cart at all.


Why You Can't Edit or Publish a Smart Cart During an A/B Test

A running Smart Cart A/B test freezes the ability to edit any carts that are in it, so nothing changes underneath the results. Carts that are not in the test stay fully editable.

  • The Cart Editor opens in read-only mode for a cart in the test, with a View-Only badge beside Configure Smart Cart in the sidebar, and no setting can be changed or saved.

  • The Your Carts table on the Smart Cart page shows an A/B badge on each cart in the test, with the tooltip This cart is part of an active A/B test and cannot be edited. End the experiment to make changes.

  • The publish toggle is disabled and reads A Smart Cart A/B test is currently running. Please end your experiment before setting a new cart live.

  • Rebuy rejects any request to enable or disable a Smart Cart while the test is running and answers Unable to enable or disable Smart Carts. with the reason An active A/B test for Smart Cart(s) is currently running.

  • A Legacy Smart Cart cannot be enabled during a Smart Cart test, and Rebuy answers Cannot enable a legacy Smart Cart while an A/B Test for Smart Carts is active.

To change any of this, end the experiment first. Every lock in this section releases as soon as the test stops.


End the test and set a winner

Ending a Smart Cart test is where you decide whether Rebuy sets the winning cart live for you. You end a test from the same A/B Testing page you created it on, which needs the Flows & A/B Testing package on your Rebuy account.

  1. On the A/B Testing page, click End Experiment on the running test.

  2. The End Experiment window opens with the prompt Choose experiment end behaviour.

  3. Choose Set Winner Live or Do Nothing.

  4. Click Stop Experiment.

Set Winner Live enables the winning cart for you, chosen against the Experiment Goal you set. If the winning variation is Do Not Show Anything there is no cart to enable, and Rebuy says No Smart Carts to enable, "Do Not Show Anything" is the winner. If no winner can be resolved, Rebuy turns on no new cart and says We could not determine a winning Smart Cart to enable. Open the Smart Cart tab to enable one. Your previous Smart Cart, the one that was live when the test started, stays on, so open the Smart Cart page to check which Smart Cart is live.

Do Nothing ends the test without setting a winner live. Open the Smart Cart page under Cart & Merchandising to see which Smart Cart is live and turn on the one you want, which is the right choice when you want to change something on the winning cart before it goes live.

Smart Cart A/B testing: the End Experiment window with the Set Winner Live and Do Nothing choices above the Stop Experiment button

Check that a cart is live after any test ends. With no Smart Cart enabled, the Smart Cart does not load on your storefront, and shoppers use your Shopify theme's native cart instead, as its drawer or its /cart page.

What happens when a Smart Cart A/B test reaches its End Date

A Smart Cart A/B test that reaches the End Date you scheduled ends on its own. Rebuy then either sets the winning Smart Cart live or switches your store back to your previous Smart Cart. Like every Smart Cart test, this needs the Flows & A/B Testing package on your Rebuy account.

Your previous Smart Cart is the Smart Cart that was live on your store when the test started, which Rebuy saves when the test starts and shows to shoppers outside the test's audience while the test runs. If that Smart Cart is a Smart Cart Legacy cart, Rebuy makes the Legacy cart live again. If no Smart Cart was live when the test started, or that Smart Cart was deleted during the test, Rebuy has no previous Smart Cart to switch back to, so shoppers see your theme's native cart.

Rebuy sets a winner live automatically only when every one of the following is true:

  • The test's Experiment Goal is Conversion Rate (cr).

  • A Smart Cart variation has a 95% or higher probability of being the best performer.

  • Every variation has at least 100 visitors.

The following table shows what all shoppers see after a Smart Cart A/B test reaches its End Date. Your previous Smart Cart means the Smart Cart that was live on your store when the test started.

Test result at the End Date

What all shoppers see after the test

A Smart Cart variation wins a Conversion Rate test with a 95% or higher probability and at least 100 visitors on every variation

The winning Smart Cart, including shoppers who were outside the test's audience

Do Not Show Anything wins a Conversion Rate test

Your previous Smart Cart, since Rebuy does not turn the Smart Cart off for your store. To run without a Smart Cart, turn it off from the Smart Cart page

A Conversion Rate test where no variation reaches 95%, or any variation has fewer than 100 visitors

Your previous Smart Cart

The test uses the Revenue goal

Your previous Smart Cart, since a Revenue goal test does not set a winner live at its End Date

No winner is set live, and there is no previous Smart Cart, because none was live when the test started or it was deleted during the test

Your theme's native cart, until you turn on a Smart Cart from the Smart Cart admin page

To put a Revenue goal winner live, turn it on from the Smart Cart admin page after the test ends, or end the test from the A/B Testing page once it has collected enough data and choose Set Winner Live.

Why a Scheduled Smart Cart Replaced My A/B Test Winner

A Smart Cart you scheduled with the Smart Cart Scheduler to start at or after the test's End Date still goes live at its start time, and it replaces whichever Smart Cart Rebuy turned on at the End Date, the winner or the Smart Cart that was live before the test, since only one Smart Cart can be live on your store at a time. A Smart Cart scheduled to start while the test is running goes live after the test ends instead, with the same result. To keep the test's outcome live, open that scheduled cart's Global Settings, open Schedule, and turn Schedule Cart off before the test's End Date, then schedule the cart again once the test has ended.


Check which Smart Cart Is Live After a Scheduled A/B Test Ends

To see which Smart Cart Rebuy turned on when a Smart Cart A/B test reached its End Date, open the test's card on the Completed tab of the A/B Testing page. The card's message explains the result and links to each Smart Cart tested and to Manage Smart Carts. The message appears only on Smart Cart tests that end on their End Date, not on tests you end yourself. Rebuy does not send an email about the outcome, and the Smart Cart page shows which Smart Cart is live now.


Frequently asked questions

Can I A/B test my theme's cart against the Smart Cart?

Yes, you can A/B test your theme's own cart against the Smart Cart, by creating a Smart Cart experiment and choosing Do Not Show Anything for one variation and a Smart Cart for the other. Shoppers in the Do Not Show Anything variation get no Smart Cart, so your theme's cart handles them for the length of the test.

What does a shopper see in the Do Not Show Anything variation?

A shopper in the Do Not Show Anything variation of a Smart Cart A/B test gets no Smart Cart. Your Shopify theme's native cart opens for them instead, as its drawer or its /cart page. This works when your theme's cart is hidden only from the Smart Cart's Custom CSS box, which does not load for these shoppers. A theme cart hidden with CSS in your theme's files, or removed from your theme, leaves these shoppers with no cart.

What happens when Do Not Show Anything wins my Smart Cart A/B test?

When Do Not Show Anything wins a Smart Cart A/B test that reaches its scheduled End Date, Rebuy switches your store back to your previous Smart Cart, the one that was live when the test started, since Rebuy does not turn your Smart Cart off for you. To act on the result and run your store on your theme's native cart, turn that Smart Cart off from the Smart Cart page under Cart & Merchandising. If no Smart Cart was live when the test started, shoppers already see your theme's native cart after the End Date.

When Do Not Show Anything wins and you end the test yourself with Set Winner Live, Rebuy shows No Smart Carts to enable, "Do Not Show Anything" is the winner, which is the expected message. Rebuy does not turn your Smart Cart off, so your previous Smart Cart, the one that was live when the test started, stays on. To act on the result and run your store on your theme's native cart, turn that Smart Cart off from the Smart Cart page under Cart & Merchandising. If no Smart Cart was live when the test started, shoppers already see your theme's native cart.

Can I edit my carts while a Smart Cart A/B test is running?

No, the carts inside a running Smart Cart A/B test cannot be edited until the test ends. The Cart Editor opens read-only with a View-Only badge, the publish toggle is disabled, and Rebuy refuses any attempt to enable or disable a Smart Cart. Carts that are not part of the test stay editable throughout.

Why is no Smart Cart live after my Smart Cart A/B test reached its End Date?

No Smart Cart is live after a Smart Cart A/B test reaches its End Date when the test ends without a winner set live and Rebuy has no saved Smart Cart to make live again, because no Smart Cart was live on your store before the test started, or the Smart Cart that was live before the test was deleted during the test. Shoppers then see your theme's native cart. The message on the test's card on the Completed tab says which of these happened, and you can turn on the Smart Cart you want from the Smart Cart page.

If the message on the card says a Smart Cart is back on but your storefront shows no Smart Cart, contact Rebuy Support with the test's name.

Can I A/B test different free shipping thresholds in the Tiered Progress Bar?

No, there is no built-in way to A/B test free shipping thresholds, because the free shipping offer itself lives in your Shopify shipping settings and the Tiered Progress Bar only reflects what Shopify is set to give. What you can test is whether the bar helps at all, by running a Smart Cart experiment with a cart that has a Tiered Progress Bar against a cart that does not.

Did this answer your question?