Head-to-Head Smart Search A/B Tests

Head-to-Head Smart Search A/B Tests is an experiment type in Rebuy's A/B Testing product that lets you run a direct, controlled comparison between Shopify Native Search and Rebuy Smart Search on your live storefront. Visitors are automatically split between the two search experiences based on your configured traffic allocation, and the platform tracks conversion performance to determine which search drives better results.

This is a zero-configuration experiment — you don't build variant content or write code. The two variants are the search products themselves: Shopify Native Search as the control and Rebuy Smart Search as the treatment.

Before creating a Head-to-Head Smart Search A/B Test, confirm that all of the following are true for your account:

If any of these conditions are not met, the Smart Search option in the experiment creation modal will be disabled and will show a status badge indicating what action is needed.

Click the badge to go directly to the step needed to complete setup.

A Head-to-Head Smart Search A/B Test splits your storefront visitors into two groups. Each group sees a different search experience, and Rebuy tracks which group converts at a higher rate.

Once a visitor is assigned to a variant, that assignment persists for the duration of their session and carries over to future visits. This ensures clean conversion attribution even after a purchase is completed.

Results are measured using Bayesian Probability to Win — the same statistical methodology used across all Rebuy A/B tests. The experiment tracks conversion rate (the percentage of visitors who complete a purchase after using search) as the primary success metric.

The following steps walk you through creating, configuring, and starting a Head-to-Head Smart Search A/B Test in Rebuy.

Before creating your experiment, confirm that your Shopify theme's search-results template is enabled. If this template is disabled, visitors assigned to the Smart Search variant may see duplicate search results during the experiment.

To create a Head-to-Head Smart Search A/B Test, navigate to Rebuy's A/B Testing product and start a new experiment.

When you open the Smart Search experiment editor, you will see two informational banners at the top of the page. Review both before proceeding.

⚠️ Additional Shopify configuration may be necessary (yellow warning)

This warning appears because your Shopify theme's search-results template may need to be enabled. See Step 1: Configure Your Shopify Theme for instructions. If you have already completed that step, this warning can be dismissed.

ℹ️ Only one Smart Search experiment can run at a time (blue info banner)

This banner explains three constraints that apply while your experiment is running:

Head-to-Head Smart Search A/B Tests have exactly two variants. You cannot add variants or change what each variant displays — only the traffic allocation between them can be adjusted.

To adjust the traffic split, update the percentage allocation for each variant. For example, you can set a 70/30 or 80/20 split if you want to limit exposure to the treatment variant during the test.

You can also set an optional display alias for each variant. Aliases only affect how the variants are labeled in your results report — they do not change the underlying search experience shown to visitors.

Once you have reviewed the warnings and configured your traffic split, you are ready to start the experiment.

When the experiment starts, the following changes take effect immediately:

To make changes to Smart Search settings during a test, you must first end the experiment manually from the A/B Testing dashboard.

To view results for a running Head-to-Head Smart Search A/B Test, navigate to A/B Testing and open the Live tab. Smart Search experiment cards are labeled with a SMART SEARCH chip so you can identify them quickly.

The results table for a Smart Search experiment includes the following metrics:

To end a running Head-to-Head Smart Search A/B Test, navigate to the A/B Testing dashboard and stop the experiment manually. Once the experiment ends, Smart Search settings return to editable and the view-only lockout is lifted.

You can create and start a new Smart Search experiment at any time, even if one is already running. However, starting a new experiment will automatically end the active one.

When you attempt to start a second experiment, Rebuy will display the following confirmation modal:

Replace Active Experiment? "[experiment name]" is currently running. Starting this experiment will automatically stop the active Smart Search experiment. Only one Smart Search experiment can run at a time. Do you want to stop the current experiment and start this one?

Click Replace & Start Experiment to end the active experiment immediately and start the new one. This action cannot be undone — the ended experiment's results will be preserved in reporting, but the test will stop collecting data.

Important: Be careful when using this action. If you replace a running experiment, the active test ends permanently. Make sure you have finished collecting data before starting a replacement.

If you have a Smart Search experiment in Scheduled status, attempting to disable Smart Search (from the Results Page or Quick View settings) will trigger a warning:

You currently have one or more scheduled experiments that involve Smart Search. If you disable Smart Search, these experiments will be moved to draft status.

Confirming the disable will move your scheduled experiment back to Draft. You will need to re-schedule it after re-enabling Smart Search.

The following limitations apply to Head-to-Head Smart Search A/B Tests in the current release:

The following capabilities are not available in this version of Head-to-Head Smart Search A/B Tests:

Note: Additional search-specific metrics — including Search Usage Rate, Click-Through Rate, Add-to-Cart Rate from Search, and Zero Results Rate — are being collected in the background and will be surfaced in a dedicated Search analytics dashboard in a future release.

Yes, running a Head-to-Head Smart Search A/B Test requires Smart Search billing to be enabled on your account. Smart Search must also be fully configured and active (toggle on) before you can create or start the experiment. If any of these conditions are not met, the Smart Search option in the experiment creation modal will be disabled. Click the status badge shown on the option to go directly to the required setup step.

No, Smart Search settings are view-only while a Head-to-Head Smart Search A/B Test is running. You cannot edit keywords, merchandising rules, results page settings, Quick View, or any other Smart Search configuration during an active test. To make changes, end the experiment first from the A/B Testing dashboard, make your updates, and then start a new experiment.

No, only one Smart Search experiment can run at a time. If you start a second experiment while one is already running, Rebuy will ask you to confirm that you want to replace the active experiment. Confirming will immediately end the running experiment and start the new one. The ended experiment's results are preserved in reporting.

Double search results in the Smart Search variant typically happen when your Shopify theme's search-results template is disabled. When this template is disabled, Shopify's native results and Smart Search results can both render on the search page at the same time. To fix this, go to your Shopify Admin, open your active theme in the Customizer, navigate to the Search page, and enable the Search Results template using the eye icon in the left sidebar. The experiment editor also shows a yellow warning with these instructions if this issue is detected.

"Needs more data" in the Prob. to Win column means the experiment has not yet collected enough traffic to reach a statistically meaningful result. Rebuy uses a Bayesian Probability to Win methodology — the same approach used across all Rebuy A/B tests. The display will update to show a probability percentage once the statistical threshold is reached. Continue running the experiment until a clear winner emerges or you have collected sufficient data to make a decision.

No, Head-to-Head Smart Search A/B Tests only support two variants: Shopify Native Search (control) and Rebuy Smart Search (treatment). Testing Smart Search against third-party search vendors is not available in the current release.