A/B Testing Overview
Rebuy's A/B Testing is a tool for optimizing conversion rates (CRO) that enables you to experiment with on-site widgets and the underlying data sources. It empowers them to make decisions based on data, ensuring a customized customer experience that drives scalable revenue growth and meets you customers' expectations for personalization.
At this time, Rebuy's A/B Testing only works with widgets but will eventually expand to work with more of Rebuy's products.
Benefits of A/B Testing
Rebuy encourages you to utilize this tool to enhance you customer experience, boost average order value (AOV), increase revenue, extend customer lifetime value (LTV), and potentially reduce customer acquisition costs (CAC).
Those who are seeking to achieve the following goals can benefit from A/B Testing:
Enhancing product discovery by offering a wider range of relevant or unique products.
Expand and refine the Ideal Customer Profile (ICP).
Improving conversion rates and enhancing the customer experience by implementing personalized language within widgets.
Evaluating the impact of Rebuy by comparing widgets against the absence of widgets.
Increasing revenue and gaining insights into the customer journey by testing widgets in different locations to boost average order value (AOV), customer lifetime value (LTV), and identify areas where customers are adding products to their cart.
Want to learn more about Rebuy features? Book a demo with our team today!
Feature List
User-Friendly Experience
The setup process for experiments is effortless, taking less than 5 minutes. Once a test is completed, the winning widget, determined based on the selected Experiment Goal during setup, will automatically be used for all incoming traffic. Accessing and reviewing reports is always convenient, directly from the A/B Testing Dashboard.
Comprehensive Dashboard
The A/B Testing dashboard provides a centralized view of ongoing, draft, and completed experiments, along with their respective metrics. From the dashboard, creating new experiments and getting started is simple. Completed tests can be accessed and reviewed at any time without the worry of them being automatically archived.
Powerful Experimentation
Test up to 9 variations of widgets against a control widget. Variations can consist of different widgets, the same widgets with different data sources, or even no widget at all. Note: If "Do Not Show Anything" is selected, certain metrics may be unavailable (e.g., comparing conversion rates across variations requires a widget for conversions).
Flexible Experiment Scheduling
Schedule experiments to start and run for a specific duration or initiate an experiment immediately, allowing it to run indefinitely until manually ended. If there's a need to conclude a scheduled test earlier, it can be easily ended with a simple click.
Limitations and FAQs
What widgets are supported by A/B Testing?
Currently, you have the option to conduct A/B testing on all widget types except for the following:
Third Party Widgets (Recharge, Tapcart, Malomo, Wonderment, etc.)
Shopify Checkout Extensions
Pre-purchase Cross-Sell
Smart Links
Klayvio Events
Attentive Events
We are actively working on enhancing our capabilities for future iterations of A/B testing to accommodate a broader range of widget types. Keep an eye out for new releases with additional A/B testing capabilities!
What are the Best Practices for A/B Testing Experiments?
Although A/B testing offers extensive customization options to meet your specific requirements, we have compiled a best practices guide to provide helpful tips and tricks for conducting a successful A/B test.
How many widget variations can I test against one another?
A/B Testing feature allows for up to 10 total variations (1 control and 9 variations).
Can I test different widget types against one another?
Yes you can test different widget types against one another.
NOTE: If a merchant installs the A/B placement widget ID in app blocks or theme code, it will not appear as a Cart Cross Sell if they later choose to add it in that capacity. Regular cross sells can be placed as app blocks and in the Smart Cart simultaneously, but the A/B testing placeholder widget cannot. To display the A/B testing placeholder widget in the cart, merchants must remove it from their theme.
How do I A/B test Post Purchase?
Currently Post Purchase A/B testing is housed within the Post Purchase flows. You can follow the Post Purchase Flow guide to set up your A/B testing experiment. A/B testing for post purchase is a bit different due to the nature of the widget type so it remains separate at this time.
How do I A/B test Smart Cart?
Currently Smart Cart is not available for A/B testing natively. You can use Experiments.js to conduct A/B testing for it. Experiments.js is an optional JavaScript package that our developers have created to make it easier for you to conduct A/B testing on various elements of your site such as Smart Cart, Widgets, and settings. With Experiments.js, you have the flexibility to use your preferred A/B testing suite like Google Optimize, VWO, Optimizely, or even your own custom solution.
Can I A/B Test Data Sources?
Yes, you can technically A/B test data sources. If you wish to do so, you will need to build out a duplicate widget and have each widget pointed to the respective data source. You can duplicate the widget by following these steps.
Duplicating Widget
Navigate to the widgets page within Rebuy Admin.
Locate the widget you wish to duplicate.
Click on the vertical ellipsis menu and click duplicate.
Once you have the duplicate widget, you can adjust your data source rules to power each widget. From there you can use those two widgets to A/B test against each other, which will ultimately A/B test each data source.
Can I A/B test different placements of the widget on my page?
Yes, if you would like to A/B test the placement of a specific widget. If you wish to do so, you will need to build out a duplicate widget and have each widget pointed to the respective dynamic placement. You can duplicate the widget by following these steps.
Duplicating Widget
Navigate to the widgets page within Rebuy Admin.
Locate the widget you wish to duplicate.
Click on the vertical ellipsis menu and click duplicate.
Setting up the widget's dynamic placement
First you will need to make sure you set up each widget as a dynamically placed widget which you can do by following the steps in the Dynamic Placement Selectors guide.
In this example, we will outline building two PDP widgets. One that will live above the add to cart button, and the other to be under the add to cart button.
Create widget A - place it above add to cart using the Dynamic Placement Selector.
Duplicate widget A - rename it widget B - place it under add the cart button using the Dynamic Placement Selector.
Once you have the two widgets built out with the dynamically placed selector configured, you can proceed to building out your A/B test.
How is the "winning" A/B widget test calculated?
The "winning" A/B test is chosen by the revenue or conversion (whichever you have chosen during the setup of the A/B test) of the page load results and not the specific performance of a widget. To learn more about the reporting structure for A/B testing, check out the Dashboards & Reporting article.
Does a Widget need to be in live mode during A/B testing?
Yes, the widget(s) should be in live mode for the A/B test. However, as a reminder the widgets are not to be installed themselves. You must use the placeholder ID for experiments. Installing the placeholder ID is simple and you can follow the steps in this guide.
How do I turn off A/B testing attributions (Rebuy-Experiment: true)
Regrettably, it's not possible to disable the A/B testing attributes, unlike the Smart Cart and Widgets, which offer that option. These attributes are essential for accurately tracking user sessions and obtaining reliable A/B testing outcomes.
