Overview
What is A/B Testing for SmartFlows?
A/B Testing for Global SmartFlows allows merchants to test multiple flow variants before making live changes. Integrated with Rebuy’s A/B Testing interface, it enables controlled experiments on Global Flow configurations without developer help.
This feature lets merchants optimize site-wide personalization (e.g., discounts, banners, triggers) based on data, not guesswork. It expands Rebuy's experimentation capabilities, creating a unified optimization layer across surfaces like Flows, Widgets, Carts, and Landing Pages.This leads to:
Data-driven optimization: Fine-tuning personalization strategies based on actual performance.
Reduced risk: Minimizing the chances of implementing changes that negatively impact the customer experience.
Expanded experimentation: Adding to the broader experimentation ecosystem within Rebuy, across all surfaces.
If you have not already done so, you will need to create your Global Smart Flow variations before beginning a Smart Flow A/B test.
How to Set Up A/B Testing for Global SmartFlows
Build Your Global Flows: First, create your global flows. A/B Testing is only available for global flows, so make sure you've set up the ones you'd like to test.
Create a New Experiment: Once your flows are ready, go to the A/B Testing tab, click Create New Experiment, and select Global Smart Flow. Name your experiment and click Add Experiment.
Set the Control and Variations: Under Control, you can choose to show nothing as the default. Then, add a variation by selecting the global flow(s) you wish to A/B test.
Adjust Display Percentages: Set the percentages for how many of your site visitors will see each variant for the duration of the test. We recommend you split your traffic evenly across variants."
Define Your Experiment Goal: Choose whether your goal is to optimize for Revenue or Conversion Rate.
Set Test Duration: Optionally, set a test duration, though you don’t need to specify a start and end time. Setting a test duration is strongly recommended/best practice.
Start Your Experiment: Once everything is set up, save your experiment and click Start Experiment. You’re all set to begin testing!
Smart Flow Experiment Goals Explained
When setting up your A/B test, you’ll need to select an Experiment Goal: Revenue or Conversion Rate. This goal will define how the success of your experiment is measured.
Revenue (default): The winner is determined by which variant (control or variation) generates the most revenue.
Conversion Rate: The winner is based on which variant (control or variation) leads to the highest percentage of order conversions.
Once the experiment has started, the goal cannot be changed. Make sure to choose the goal that best aligns with your optimization objectives before starting the test.
Start and End Time Explained (Optional)
To specify a test duration, go to the Test Duration section. You can choose to either:
Manual Control: If you prefer to start and end the experiment manually, there's no need to set a start or end time.
Automated Timing: If you want the experiment to start and end automatically, use the Start Date and Start Time selectors to schedule when the test will begin. Similarly, set the End Date and End Time to specify when the test should automatically conclude (the end time must be after the start time).
You can adjust the scheduled start or end times at any point before the experiment begins or after it has ended.
Starting an Smart Flow A/B Test Experiment
Starting from the Edit Screen:
If you've saved and remained on the edit screen, you can start the experiment immediately.
Look for the "Start Experiment" button near the top right-hand side of the experiment editor (in the main heading).
Click "Start Experiment". A confirmation pop-up will appear.
Confirm by selecting "Start Experiment" again, and your A/B test will go live.
Starting from the Dashboard:
If you've saved and exited the edit screen, you can start the experiment from the Drafts tab in the A/B Testing dashboard.
Locate the ellipsis menu (three dots) next to the word "Draft" for the experiment you want to start.
From the menu, click "Start Experiment". A confirmation pop-up will appear.
Confirm by selecting "Start Experiment" again, and your A/B test will go live.
Ending a Smart Flow A/B Test
Scheduled End Time: If you set a scheduled end time for your experiment, it will automatically conclude according to your configured settings.
Ending or Editing the Experiment: If you want to end the experiment early or adjust the end time, you can do so using the vertical ellipsis menu (three dots) next to the experiment on the A/B testing dashboard. From there, you can either immediately end the experiment or edit it to adjust the scheduled end time.
Results: Once the experiment ends, the results will be available under the "Completed" tab on the A/B testing dashboard. The variation that "won" the test will be the one that goes live.
⚠️ Important: If the winning variation was set to "Do Not Show Anything," no Smart Flow will be running on the page.
Tracking Your Smart Flow Experiments
Once your A/B test experiment is active, you can track its performance:
Finding Active Experiments: Go to the "A/B Testing" page under "Manage Your Experiments". Your experiment will be listed in the "Active" table.
Managing Experiments: Use the vertical ellipsis menu (three dots) next to your experiment to either end it immediately or edit the end time if needed.
Performance Metrics
Metrics Visibility: Performance metrics are displayed in the summary blocks of each experiment, both under the "Active" and "Completed" tabs. Here you can see traffic, visitors, conversations, revenue, and amount per user. You can read more about the definitions in our article below.
Limitations of A/B Testing for SmartFlows
Access: A/B Testing for SmartFlows is available through the A/B Testing interface in Rebuy, not within the Smart Flows editor (for V1 users). You can select Global Flows from a dropdown list in the A/B Testing interface.
Control Setup: The "Control" dropdown defaults to “Do Nothing,” serving as the baseline for comparison. Once an A/B test is active, the selected Global Flow(s) are live only for the experiment, and regular Global Flow publishing is disabled to avoid conflicts.
Editing Restrictions:
You cannot edit, toggle, or replace active Global Flows during an A/B test.
To modify or republish a flow, the active test must be stopped.
Single Test Limit: Only one A/B test (Smart Cart or Smart Flow) can run at a time. If you want to start a new test, the current test must first be stopped, and a confirmation will prompt you to disable the active test.
Out of Scope for V1:
Link-Based Flows: Testing of Link-Based Flows is not supported in A/B Testing.
Global Flow vs. Link-Based Flow: You cannot test a Global Flow against a Link-Based Flow.
Creating/Editing A/B Tests in Smart Flows UI: For V1, A/B tests cannot be created or edited directly within the Smart Flows UI. A hybrid UX flow for this is planned for a future sprint.
FAQs
Can I run multiple A/B tests at once?
No, only one A/B test (Smart Cart or Smart Flow) can run at a time. You must stop the active test before starting a new one.
Can I edit Global Flows during an A/B test?
No, editing, toggling, or replacing active Global Flows is not allowed while an A/B test is running. You must stop the test before making any changes.
Can I test Link-Based Flows with A/B Testing?
No, A/B testing for Link-Based Flows is not currently supported. Only Global Flows can be tested.
Can I create or edit A/B tests from within the Smart Flows UI?
No, A/B tests must be managed through the A/B Testing interface. Editing or creating tests directly within the Smart Flows UI is not available in V1. This functionality is planned for a future update.
Can I run both a Smart Cart and a Smart Flow A/B test at the same time?
No, only one A/B test can run at a time, whether it’s for Smart Cart or Smart Flow. You must stop the active test before starting a different one.