Overview
A/B Testing with Rebuy provides an intuitive way to test different versions of your store to improve engagement and revenue metrics. With a quick setup process, requiring less than 5 minutes, you can easily determine the best-performing widgets for your store. Once an experiment concludes, the winning widget—determined by the Experiment Goal chosen during setup—will automatically be deployed for all incoming traffic. Additionally, experiment reports can be conveniently accessed and reviewed directly from the A/B Testing Dashboard.
Building a Checkout Offer A/B Test
Step 1: Create the Checkout Widget or Content Block
Before setting up an A/B test experiment, you need to create at least two Rebuy checkout widgets/content blocks (or one widget if testing against a "no widget" scenario). These widgets are the elements you will test against each other in the experiment.
If you haven’t created any checkout widgets yet, refer to our documentation for instructions on building them.
If you haven’t created any content blocks yet, refer to our documentation for instructions on building them.
Checkout widgets are exclusively available to Shopify Plus plans due to Shopify's limitations. While all merchants can access content blocks, only Shopify Plus merchants have the ability to place them in the checkout.
Step 2: Access the A/B Testing Feature
Navigate to "A/B Testing": Open the main left-hand Rebuy menu and select the “A/B Testing” option.
Create a New Experiment:
Click the “Create New Experiment” button.
In the "New A/B Test" modal, enter a descriptive experiment name in the "Name" box. This helps you track and identify the experiment later.
Choose "Checkout Offers" as the "Experiment Type."
Click “Add Experiment” to proceed to the experiment editor.
Step 3: Set Control and Variations
On the Experiment settings page, locate the "Select Control and Variations" section.
Choose your control widget, which serves as the baseline for comparison.
Add up to 9 widget variations that you want to test against the control. You must select at least one variation.
Set the display percentage for each widget (both the control and variations). This determines how often each widget will be shown to users during the experiment.
Step 4: Select the Experiment Goal
Scroll down to the "Experiment Goal" section.
Choose either "Revenue" (default) or "Conversion Rate" as the primary metric for determining the winner:
Revenue: The variation that generates the most revenue wins. This goal is ideal if you’re optimizing for total income.
Conversion Rate: The variation with the highest percentage of order conversions wins. This goal is recommended if you’re focusing on improving the number of completed purchases.
Step 5: Set Start and End Time (Optional)
You have the option to set a specific test duration or leave it under manual control:
Scroll to the "Test Duration" section.
Use the "Start Date" and "Start Time" selectors to schedule when the experiment begins.
Set the "End Date" and "End Time" to automatically end the experiment at a specific time.
You can modify these settings anytime before the experiment starts or concludes.
If you prefer to manually start and stop the experiment, skip this step.
Step 6: Save the Experiment
Once all required elements have been configured, the “Save Experiment” button will become active. Click it to finalize your experiment setup.
Step 7: Obtain the Placeholder ID
Your experiment is now in the "Draft" phase. Before starting it, you need to install a widget placeholder in your store:
On the right side of the editor, you’ll find a "Placeholder ID" value.
Use this Placeholder ID as the widget ID when installing the placeholder widget on your store’s checkout page. This placeholder ensures that the A/B test runs correctly.
Important:
Do not directly install the original widget IDs for your control and variations. Doing so will cause them to be displayed simultaneously, invalidating your A/B test.
The Placeholder ID is essential for the test to function correctly by dynamically showing the appropriate widget variation.
Step 8: Install the Placeholder ID
Installing the Placeholder ID via Shopify's checkout editor
Open the checkout editor in your Shopify portal.
Add Rebuy Engine app block.
Add the Placeholder ID directly into the app block.
For more detailed instructions on installing via app blocks, refer to Rebuy’s app blocks help documentation.
Please note: If you install the placeholder ID in your checkout and your A/B test is not yet live, an error message will appear stating: Invalid Widget ID (incorrect type or possibly deleted): Verify your widgets before updating extension in the Checkout Editor
Once you start your A/B test and refresh the checkout editor page, the error message will disappear, and a new one will appear stating: Experiment: The selected variant will display here
Starting an Experiment
Option 1: Start from the Edit Screen
If you remain on the experiment’s edit screen after saving it, you can start the experiment immediately.
Locate the "Start Experiment" button at the top right-hand side of the experiment editor (in the main heading).
Click "Start Experiment." A pop-up confirmation will appear.
Click "Start Experiment" again to confirm, and your A/B test will go live.
Option 2: Start from the Dashboard
If you leave the edit screen, you can start the experiment from the Drafts tab on the A/B Testing dashboard.
Find the experiment you want to start and click the vertical ellipsis (three dots) next to the "Draft" label.
Select "Start Experiment" from the menu. A pop-up confirmation will appear.
Click "Start Experiment" again to confirm, and your A/B test will go live.
Ending a Checkout Offer A/B Test
Scheduled End Time
If you configured a scheduled end time during setup, the experiment will automatically conclude based on that schedule.
Manually Ending the Experiment
If you wish to end the experiment early, go to the Active tab on the A/B Testing dashboard.
Click the vertical ellipsis (three dots) next to the active experiment.
Select "End Experiment" to immediately stop the test, or "Edit" to adjust the scheduled end time.
After the Experiment Ends
Once the experiment concludes, the results will be available under the Completed tab on the A/B Testing dashboard.
The winning widget configuration will automatically replace the placeholder in your store, making it permanently visible to customers without any further action required.
Tracking Your Experiments
Monitoring Active Experiments
After an experiment starts, you can view it under the Active tab on the A/B Testing page within Manage Your Experiments.
Use the vertical ellipsis menu to end the experiment immediately or edit it to modify the scheduled end time if desired.
Viewing Performance Metrics
Summary performance metrics for both active and completed experiments are visible within their respective tabs.
Metrics like revenue, conversion rate, and engagement help you assess how well each variation performs against the control.
Summary and Best Practices
Starting the Experiment: Choose either the edit screen or the dashboard to launch your experiment. Both methods offer a straightforward way to get your test live.
Managing and Ending the Experiment: Utilize the scheduling features or manually end the test using the dashboard tools.
Automatic Deployment of Winning Widget: Rebuy’s system ensures that once a test concludes, the winning widget is deployed automatically, optimizing your store’s performance seamlessly.
FAQs
Q: Can I A/B test content blocks with Rebuy?
A: Yes, you can A/B test content blocks, but it’s important to be aware of certain limitations and requirements based on your Shopify plan.
Q: Can I use the A/B checkout offer tests if I am not on Shopify Plus and what are the limitations?
A: Yes, however A/B testing checkout offers is more limited for non-Shopify Plus merchants. This is due to Shopify only allowing Shopify Plus users access to add content to their checkout pages.
Shopify Plus Merchants
If you are a Shopify Plus merchant, you have the following capabilities:
Add Checkout Extension Blocks: You can integrate product recommendations, images, and other content directly into your checkout.
Enhanced Customization: Customize content for the checkout, order status page, and thank you page with checkout extensions content blocks.
Limitations: While you can add content blocks to the checkout and thank you page, product recommendations cannot be added to the order status or thank you page.
Non-Shopify Plus Merchants
If you are not on Shopify Plus, your options for A/B testing checkout offers are more limited:
No Access to Checkout Product Recommendations: You cannot add product recommendation blocks to your checkout.
Content Block Customization: You can still enhance your order status and thank you pages by adding images or content via content blocks, but these customizations are limited compared to Shopify Plus.
Q: How can I validate which Widget ID is rendering on the Checkout page?
A: To confirm which Widget ID is rendering on the Checkout Page, open your browser's Developer Tools, then click on the Console tab. You should then see a validation message which contains the Experiment ID and Widget ID. The Selected ID is the Rebuy Widget ID that is rendering on the page via the experiment:
