Overview
A/B Testing Smart Cart provides an intuitive way to test different versions of your cart to improve conversion and revenue metrics. With a quick setup process, requiring less than 5 minutes, you can easily determine the best-performing cart for your store. Once an experiment concludes, the choice will be given to automatically enable the winning Smart Cart, or ‘Do Nothing’, allowing you to enable it manually, yourself, at a later time.
Additionally, experiment reports can be conveniently accessed and reviewed directly from the A/B Testing Dashboard.
Important
A/B Testing is only available for Smart Cart V2 carts. Legacy carts are not supported in this feature. If you are using a legacy cart, you’ll need to upgrade to a Smart Cart V2 to utilize A/B testing
Building a Smart Cart A/B Test
Step 1: Create Smart Carts for Testing
What to Do: Create at least two Smart Carts to use as variants in your A/B test. If you’re testing against a "no cart" scenario, create one cart.
Important Notes:
Once a test is live, you cannot edit the carts used in the experiment until it ends. Ensure the carts are fully configured before proceeding.
Need help creating a Smart Cart? Refer to our Smart Cart Documentation for detailed instructions.
Step 2: Access the A/B Testing Feature
Navigate to A/B Testing:
Open the main Rebuy menu on the left-hand side and select "A/B Testing."
Create a New Experiment:
Click the "Create New Experiment" button.
In the "New A/B Test" modal:
Enter a descriptive name for the experiment in the "Name" field.
Choose "Smart Cart" as the "Experiment Type."
Click "Add Experiment" to open the experiment editor.
Step 3: Set Control and Variations
Locate the "Select Control and Variations" section in the Experiment settings page.
Choose the Control Cart:
This will serve as the baseline for comparison.
Add Variations:
Add up to 9 cart variations to test against the control. At least one variation is required.
Set Display Percentages:
Allocate a percentage for how often each cart (control and variations) will appear to users during the experiment.
Preview Variations:
You can preview all variation carts from this step (except for the "Do Not Show Anything" option).
Step 4: Select the Experiment Goal
Scroll down to the "Experiment Goal" section.
Choose either "Revenue" (default) or "Conversion Rate" as the primary metric for determining the winner:
Revenue: The variation that generates the most revenue wins. This goal is ideal if you’re optimizing for total income.
Conversion Rate: The variation with the highest percentage of order conversions wins. This goal is recommended if you’re focusing on improving the number of completed purchases.
Step 5: Set Start and End Time (Optional)
Go to the "Test Duration" section.
Choose test timing:
Set a Start Date and Start Time to schedule the beginning of the experiment.
Set an End Date and End Time to automatically conclude the experiment.
Manual Option: Skip this step if you prefer to start and stop the experiment manually.
Adjustments: These settings can be changed any time before the experiment starts or ends.
A/B tests cannot be scheduled to start more than 30 days in advance or scheduled to end more than 30 days after the start date.
If you prefer to manually start and stop the experiment, skip this step.
Step 6: Save the Experiment
Once all required elements have been configured, the “Save Experiment” button will become active. Click it to finalize your experiment setup.
Starting an Experiment
Option 1: Start from the Edit Screen
If you remain on the experiment’s edit screen after saving it, you can start the experiment immediately.
Locate the "Start Experiment" button at the top right-hand side of the experiment editor (in the main heading).
Click "Start Experiment." A pop-up confirmation will appear.
Click "Start Experiment" again to confirm, and your A/B test will go live.
Option 2: Start from the Dashboard
If you leave the edit screen, you can start the experiment from the Drafts tab on the A/B Testing dashboard.
Find the experiment you want to start and click the vertical ellipsis (three dots) next to the "Draft" label.
Select "Start Experiment" from the menu. A pop-up confirmation will appear.
Click "Start Experiment" again to confirm, and your A/B test will go live.
Important
While a Smart Cart A/B test is running, the ability to enable 'new' carts from the Smart Cart tab in the admin will be disabled
Any cart selected as an experiment variation will remain in a 'view only' state so to retain the parameters set during the set up of the A/B test
Ending a Smart Cart A/B Test
Scheduled End Time
If you configured a scheduled end time during setup, the experiment will automatically conclude based on that schedule. Upon end date, all variants will be disabled. Manually enable your desired Smart Cart from the Smart Cart tab in the Rebuy admin.
Manually Ending the Experiment
If you wish to end the experiment early, go to the Active tab on the A/B Testing dashboard.
Click the vertical ellipsis (three dots) next to the active experiment.
Select "End Experiment" to immediately stop the test, or "Edit" to adjust the scheduled end time.
You will be given two options:
Set Winner Live: Based on the KPI parameters of your experiment (Revenue vs Conversion) this selection will automatically set the 'winning' variation live
Do Nothing: No variation will be set live; navigate to the Smart Cart tab in the admin to enable the cart of your choice
If a live experiment is deleted, 'Do Nothing' behaviour will automatically apply
If a Smart Cart included in a live experiment is deleted, 'Do Nothing' behaviour will automatically apply
After the Experiment Ends
Once the experiment concludes, the results will be available under the Completed tab on the A/B Testing dashboard.
All functionality in terms of enabling/editing Smart Cart from the Rebuy admin will be restored
Tracking Your Experiments
Monitoring Active Experiments
After an experiment starts, you can view it under the Active tab on the A/B Testing page within Manage Your Experiments.
Use the vertical ellipsis menu to end the experiment immediately or edit it to modify the scheduled end time if desired.
Viewing Performance Metrics
Summary performance metrics for both active and completed experiments are visible within their respective tabs.
Metrics like revenue, conversion rate, and engagement help you assess how well each variation performs against the control.
Summary and Best Practices
Starting the Experiment: Choose either the edit screen or the dashboard to launch your experiment. Both methods offer a straightforward way to get your test live.
Managing and Ending the Experiment: Utilize the scheduling features or manually end the test using the dashboard tools.
FAQs
Q: Can I A/B test my current cart (non-Rebuy) against Smart Cart?
A: Yes, by utilizing the 'Do Not Show Anything' option in the variation selection dropdown, and selecting a Smart Cart as the second variation.
Q: Will any discounting logic I have created via Functions in my Smart Cart settings still work?
A: Yes, Functions discounting in Rebuy is designed to be tied to specific cart IDs, which remain prevalent throughout the duration of the experiment. All Smart Cart settings and configurations are designed to run as 'normal' when presented to their segmented cohort.
Q: Can I edit my carts while the experiment is live?
A: No, once an experiment is live, the carts used in the test cannot be edited until the experiment ends. Ensure all configurations are finalized before starting the test. You will, however, be able to edit any cart not included in the experiment.