Skip to main content

A/B Testing | Dashboards & Reporting

Read the A/B Testing Experiments dashboard: its Live, Scheduled, Draft, Completed, and Archive tabs, its metric columns, and how Rebuy picks a winner.

Written by Christian Sokolowski

A/B Testing Experiments Dashboard

The A/B Testing Experiments dashboard shows how each of your experiments is performing, comparing your control against every variation you chose to test. It covers all Rebuy experiment types, including Widget, Smart Cart, Checkout Offer, General, Global Smart Flow, and Smart Search experiments. If you have not set up an experiment yet, follow the install instructions to get started.

The dashboard organizes your experiments across five tabs: Live for experiments running now, Scheduled for experiments set to start on a future date, Draft for experiments you created but have not launched, Completed for experiments that have ended (grouped by how strong the result is), and Archive for completed experiments you have set aside. This article explains each tab and every metric column you will see.

The metrics on this dashboard are measured at the page level, not the individual widget level, so they reflect how each variation affects the entire shopping experience. To review performance for a specific widget instead, open your Performance Report page from the Rebuy Admin Reports. Revenue and order figures are scoped to Rebuy-tracked orders, so they are built to compare variants against each other and will not match your Shopify store totals.


Live Experiments

Variation Names

These names are the aliases you chose during the installation process. If you did not set any specific alias for each of the widgets or control, then it will default to the widget name.

A/B Testing Experiments dashboard showing the Variation Names column

(Image above: Experiments dashboard row showing the Variation Names column, where each variant is labeled with the alias set during setup, for example, Control and Best Sellers, defaulting to the widget name when no alias was entered.)


Traffic

This is the allocated traffic you elected to assign to each variation type. The traffic percentages you have chosen for each variation type will be assigned randomly, similar to flipping a coin. As a result, the traffic distribution among the variations may not be exactly equal. This approach is implemented to prioritize the speedy loading of the widgets. By randomizing the traffic allocation, the widgets can be loaded quickly and efficiently, providing a seamless user experience.

A/B Testing Experiments dashboard showing the Traffic column

(Image above: Experiments dashboard row showing the Traffic column, where each variant displays its allocated traffic percentage. For example, 50% to the control and 50% to the variation).


Visitors

The Visitors column counts the visitors or views recorded for each variation, tracked with a cookie set on page load. Once a visitor is shown a variation, Rebuy stores that assignment in an _r_experiment_<experiment ID> cookie that lasts 14 days, so the visitor keeps seeing the same variation across separate visits inside that window rather than only for one browsing session. Starting a new Shopify cart refreshes the 14-day window and does not reassign the variant. The assignment is lost only if the visitor clears their cookies or browses in a private window, where each new session starts with a fresh cookie. This cookie-based assignment is what keeps the experience consistent for each visitor and makes the per-variation visitor counts a valid basis for comparison.

When an experiment uses Audience Targeting, the Visitors column counts only the shoppers who matched the experiment's audience and were entered into the test. Shoppers outside the audience are never assigned a variation and never appear in the Visitors count, so an audience-targeted experiment shows fewer visitors than the same experiment run on Everyone. See How to Target an A/B Test to a Specific Audience.

A/B Testing Experiments dashboard showing the Visitors column

(Image above: Experiments dashboard row showing the Visitors column, where each variant displays its total visitor count from cookie-based page-load tracking. For example, 8,196 visitors for the control and 8,254 for the variation).


Conversions

The Conversions column in the Experiments dashboard shows the number of orders placed by shoppers who were shown a given variant, counted only when Rebuy's experiment tag was recorded at checkout. It is a raw order count, which is different from the Conversion column (the conversion rate percentage) shown elsewhere in the same table.

Because Rebuy counts only orders that carried the experiment tag at checkout, the Conversions number tracks a scoped subset of your store's orders and will be lower than your Shopify order count. This is by design: counting only tagged orders is what makes the comparison between variants valid. Use Conversions to compare one variant against another within the same experiment, not to reconcile against your Shopify store totals.

A/B Testing Experiments dashboard showing the Conversions column, which is different than the Conversion column

(Image above: Experiments dashboard variant rows with the Conversions column highlighted)


Conversion

The "Conversion Rate" option is used to determine the winning variation in an A/B test based on which variation, including the control, achieves the highest percentage of order conversions. In other words, it assesses the success of each variation by comparing the conversion rates, which represent the proportion of visitors who complete a desired action.

The conversion rate is based on page views of the widgets or controls. Not the individual widget conversion. If you want to see widget conversion rates, visit the performance reporting section. The "winning" widget is chosen based off of the page load conversion that is shown on the A/B testing dashboard and not the specific widget that made the most conversions during the A/B test.

Conversion rate is measured differently on Checkout Offer experiments that run on the Thank You page or Order Status page. Those pages load only after checkout is complete, so the order counted against each variant is the order the shopper had already placed before the variant appeared. See A/B Testing Guide: Checkout, Thank You Page, and Order Status Page Offers for details on reading those results.

A/B Testing Experiments dashboard showing the Conversion column

(Image above: Experiments dashboard variant rows with the Conversion column highlighted, showing the dollar total and the per-visitor value beneath it.)


Total Revenue

The Total Revenue column in the Experiments dashboard shows the total store revenue from orders attributed to a given variant, including revenue beyond what Rebuy generated. It reflects the overall business impact of that variant, not only the revenue Rebuy drove. Beneath each figure, the dashboard also shows a per-visitor value, which is the variant's total revenue divided by its visitor count.

For example, a variant with $19,232.32 in Total Revenue across 8,196 visitors shows $2.35 per visitor ($19,232.32 ÷ 8,196 = $2.35).

A/B Testing Experiments dashboard showing the Total Revenue column

(Image above: Experiments dashboard variant rows with the Total Revenue column highlighted, showing the dollar total and the per-visitor value beneath it.)

The way revenue is calculated differs depending on the type of experiment you're running:

  • Widget experiments display Rebuy-attributed revenue — the revenue from items added through Rebuy-powered recommendations during the experiment period.

  • Smart Cart experiments display total order revenue — the full cart value for sessions where the Smart Cart was active, not just Rebuy-attributed items.

This means the revenue figures across different experiment types are not directly comparable. If you're reviewing results from a widget test and a Smart Cart test side by side, keep in mind they are measuring different things.

Rebuy Revenue

The Rebuy Revenue column in the Experiments dashboard shows the revenue Rebuy generated for a given variant. This is the value attributed to Rebuy for the orders tracked in that variant, not the full value of every store order, so it is a subset of the variant's Total Revenue. Beneath each figure, the dashboard also shows a per-visitor value, which is the variant's Rebuy Revenue divided by its visitor count.

For example, a variant with $1,211.40 in Rebuy Revenue across 8,196 visitors shows $0.15 per visitor ($1,211.40 ÷ 8,196 = $0.15).

A/B Testing Experiments dashboard showing the Rebuy Revenue column

(Image above: Experiments dashboard variant rows with the Rebuy Revenue column highlighted, showing the dollar total and the per-visitor value beneath it.)


Rebuy AOV

The Rebuy AOV column in the Experiments dashboard shows the average order value Rebuy attributes to each variant, calculated as Rebuy Revenue ÷ Conversions. Rebuy Revenue is the value attributed to Rebuy for the orders tracked in that variant, not the full value of every store order, and Conversions is the count of those tracked orders.

For example, a variant with $1,524.28 in Rebuy Revenue across 98 conversions shows a Rebuy AOV of $15.55 ($1,524.28 ÷ 98 = $15.55).

Because Rebuy AOV is calculated from Rebuy-tracked orders only, it will differ from your Shopify store AOV. Use it to compare the average order value between variants in the same experiment, not as a store-wide benchmark.

A/B Testing Experiments dashboard showing the Rebuy AOV column

(Image above: Experiments dashboard variant rows with the Rebuy AOV column highlighted, showing the info tooltip that reads "Rebuy AOV = Rebuy Revenue ÷ Conversions.")


Segment Live Experiment Results by Device, UTM Source, or Visitor Type

A/B test segments are in beta. Rebuy turns on the Segment row for each store individually during the beta, so your Live experiment cards may not show the Device, UTM Source, and Visitor dropdowns yet. If you don't see a Segment row, contact Rebuy Support to ask for segments on your store. Segment data is collected starting from the day segments are turned on for your store, so traffic from before that day doesn't appear in any segment.

The Live tab of the Experiments dashboard has a Segment row above the results table, holding a Device dropdown, a UTM Source dropdown, and a Visitor dropdown. Picking a value from one of them re-reads the same experiment's results for only that slice of traffic, so you can tell whether a variant won across your whole audience or only inside one part of it. A test that looks flat overall is often a real gain for one group and a loss for another, and the blended number hides both. Rebuy enables segments per account while the feature is in beta, so if your experiment cards have no Segment row, contact Rebuy Support to have it turned on for your store.

Choosing a segment changes what the table reports and never changes who enters the experiment. Rebuy assigns every visitor to a variant with a random traffic split that ignores device, visitor type, and traffic source, so setting Device to Desktop shows you how desktop visitors performed while mobile visitors keep entering the test and keep being counted in the unsegmented view.

Segments are on the Live tab. Support for the Completed and Archive tabs is planned and not yet released.

A/B Testing segment controls, the Segment row on a Live experiment card with the Device, UTM Source and Visitor dropdowns above the results table

How do I apply a segment to a live experiment?

To apply a segment to a live experiment on a store that has segments turned on during the beta, open the Live tab, find the experiment, and pick a value from the Device, UTM Source, or Visitor dropdown in the Segment row. The table updates immediately, with no reload and no effect on the running test.

Each dropdown lists its values with the visitor count recorded for that value and the share of the experiment's total visitors it represents. A value that has recorded no traffic yet is greyed out, reads No data yet, and cannot be selected. Once you pick a value, a chip appears in the Segment row naming the dropdown and the value, such as Device: Desktop, with an X to clear it, and a note above the table repeats which value is applied along with its count and share.

That note also states that the winner and the probability to win reflect all traffic rather than the segment you selected, because Rebuy does not recalculate a winner or a probability inside a segment. Read a segmented view for each variant's visitors, conversions, conversion rate, and revenue figures, and take the winner call and the confidence number from the unsegmented view.

Rebuy reports one segment value at a time and has no combined view, so you can see how mobile visitors performed and how email visitors performed but not how mobile visitors who arrived from email performed. If you need a decision about one specific combination of device and channel, run a dedicated experiment for it.


Scheduled Experiments

The Scheduled tab of the Experiments dashboard lists experiments you have set to start automatically at a future date. A scheduled experiment has not started collecting data yet, so its card shows the variants, traffic split, and start and end dates you configured, but performance columns such as Conversions and Total Revenue show a dash until the experiment goes live.

Open the Scheduled tab from Experiments to review or change a test before its start date. From a scheduled experiment's actions menu (the three-dot icon) you can do the following:

  • Edit the experiment settings, variants, or schedule.

  • Preview a variant on your storefront (available for Widget, Smart Cart, General, and Global Smart Flow experiments).

  • Start Experiment to launch it immediately instead of waiting for the scheduled date.

  • Delete the experiment.

Only one Smart Cart, Global Smart Flow, or Smart Search experiment can run at a time. If you schedule one of these while another of the same type is already active, Rebuy warns you that the active experiment will automatically end when the scheduled one starts.

When the Scheduled tab is empty, it reads "No scheduled experiments" with the note "Schedule an experiment to start automatically at a future date."

(Image above: The Experiments dashboard with the Scheduled tab selected, showing a scheduled experiment card with its start and end dates and dashes in the performance columns.)


Draft Experiments

The Draft tab of the Experiments dashboard lists experiments you have created but not yet launched or scheduled. A draft is a work in progress: it has no start date and has collected no data, so its performance columns stay empty until you start it.

Open the Draft tab from Experiments to finish setting up a test. From a draft's actions menu (the three-dot icon) you can do the following:

  • Edit the draft to finish configuring its variants, traffic split, and goal.

  • Preview a variant on your storefront (available for Widget, Smart Cart, General, and Global Smart Flow experiments).

  • Start Experiment to launch it now, or open it to set a schedule.

  • Delete the draft.

When the Draft tab is empty, it reads "No draft experiments" with the note "Drafts are experiments you've created but haven't launched yet."

(Image above: The Experiments dashboard with the Draft tab selected, showing a draft experiment card with editable variants and no performance data.)


Completed Experiments

The Completed tab of the Experiments dashboard lists experiments that have ended. Rebuy groups each completed experiment by how strong its result is, so you can see at a glance which tests produced a winner and which need another run. Each experiment appears as a card you can expand to see the full variant comparison.

How are completed experiments grouped?

Rebuy sorts every completed experiment into one of four groups based on the winning variant's probability of being the best performer, shown in the Prob. Best column. The following table explains each group and what qualifies an experiment for it:

Group

What it means

Requirement

Clear Winners

A variant is a statistically clear winner you can deploy with confidence.

The best variant has a 95% or higher probability of being the best performer.

Strong Signals

A variant is very likely the best, but has not yet reached the clear-winner bar.

The best variant's probability is between 70% and 95%.

Needs More Data

No variant has reached a confident result yet.

The best variant's probability is below 70%, or there is not enough data to calculate one.

No Data Collected

The experiment recorded no usable results.

No variant received any visitors, or the experiment had no valid test variant.

What does a completed experiment card show?

A completed experiment card summarizes the result before you expand it. Each card includes the following:

  • A result badge: "Clear Winner: [variant name]" for a clear winner, "Strong Signal: [variant name]" for a strong signal, "Not Enough Data", or "No Data".

  • A confidence badge for clear winners and strong signals, such as "95% Confident", showing the winning variant's probability of being the best performer.

  • A revenue advantage badge when the winner earned more than the control, such as "+$18,893 Revenue Advantage", showing how much more revenue the winning variant drove than the control.

  • A question and answer summary: the experiment name as the question, and a plain-language result such as "[Variant] increased conversion rate by +8% with 95% confidence."

  • A summary line showing the total visitors, the conversion-rate lift over the control, and how many days the experiment ran, such as "155,644 visitors · +8% lift · 19 days."

Select Show Details on a card to expand the full variant comparison table, which lists each variant's Conv. Rate, Conversions, Visitors, Total Revenue, Rebuy Revenue (hidden for Smart Cart experiments), Rebuy AOV, and Prob. Best. The winning variant's row is highlighted, and each metric shows its change compared with the control.

From a completed experiment you can select Archive to move it to the Archive tab, or use the actions menu to Delete it.

A Clear Winners card on the Completed tab, expanded to show the Control and winning variant rows with conversion rate, revenue, and Prob. Best columns.

What does the message on a scheduled Smart Cart test's card mean?

The message on a Smart Cart test's card on the Completed tab tells you which Smart Cart Rebuy turned on when the test reached its scheduled End Date. It appears only on Smart Cart experiments that ended on their End Date, and not on experiments you ended yourself, on other experiment types, or on Smart Cart tests that ended before Rebuy added the message. Each message gives the dates the test ran, why Rebuy chose that Smart Cart, and the Smart Carts tested, and it stays on the card, since it cannot be dismissed.

The following table lists the titles of the message on a scheduled Smart Cart test's card, with a column for the test result that produces each title and a column for the Smart Cart that is live for all shoppers afterward.

Message title

Test result

Smart Cart live afterward

Set live for all shoppers: [cart name]

Conversion Rate goal, and a Smart Cart variation reached a 95% or higher probability of being the best performer with at least 100 visitors on every variation

The winning Smart Cart

No winner set live. [cart name] is back on

Conversion Rate goal, Do Not Show Anything won or no variation reached 95% with at least 100 visitors on every variation, and a Smart Cart was live before the test

The Smart Cart that was live when the test started

Test ended. [cart name] is back on

Revenue goal, any result, and a Smart Cart was live before the test

The Smart Cart that was live when the test started

No winner set live. No Smart Cart is on, or Test ended. No Smart Cart is on

No winner set live, and there is no Smart Cart from before the test to bring back, because none was live when the test started or it was deleted during the test

No Smart Cart, so shoppers see your theme's native cart

Each Smart Cart name in the message opens that cart's settings, and the Manage Smart Carts link opens your list of Smart Carts, where you can turn on a different cart. Rebuy does not email you about the outcome, so this message is where you check which Smart Cart is live after a scheduled Smart Cart test ends. How Rebuy chooses the Smart Cart is covered in A/B Testing | Smart Cart.

Why does my completed experiment say "Not Enough Data"?

Your completed experiment says "Not Enough Data" because none of its variants reached a 70% probability of being the best performer, so Rebuy could not name a statistically significant winner. Rebuy shows the message "Not enough data has been collected to determine a statistically significant winner. Consider extending the test duration or increasing traffic." To get a conclusive result, run the experiment again for a longer period or send more traffic to it so each variant collects enough visitors and orders.


Archived Experiments

The Archive tab of the Experiments dashboard stores completed experiments you have moved out of the Completed tab to keep your active results focused. Archiving does not delete an experiment or its data; it moves the experiment into a separate, searchable list.

To archive an experiment, open the Completed tab and select Archive on any completed experiment card. To move an experiment back, open its actions menu on the Archive tab and select Unarchive, which returns it to the Completed tab.

The Archive tab displays experiments in a table with the following columns:

Column

What it shows

Experiment

The experiment name and ID. Select a row to expand its full variant comparison.

Type

The experiment type, such as Widget, Smart Cart, Checkout Offer, General, Smart Flow, or Smart Search.

Signal

The result signal: Clear Winner, Strong Signal, Not Enough Data, or No Data.

Archived

The date the experiment was archived.

Above the table, you can narrow the list with the following controls:

  • A Search archived... box that matches the experiment name and type.

  • An All signals filter with the options Clear Winner, Strong Signal, Not Enough Data, and No Data.

  • An All types filter with the options Widget, Checkout Offer, Smart Cart, General, Smart Flow, and Smart Search.

  • A sort control with the options Date archived (newest), Date archived (oldest), Experiment name (A–Z), Experiment name (Z–A), and Experiment type.

Expand any row to see the same variant comparison table shown on completed experiment cards. From a row's actions menu you can Unarchive the experiment or Delete Permanently, which removes the experiment and its data and cannot be undone.

When the Archive tab is empty, it reads "No archived experiments" with the note "Archive experiments from the Completed tab to keep your view focused. Click 'Archive' on any completed experiment card to move it here."

(Image above: The Archive tab showing an archived Smart Cart experiment with a Not Enough Data signal, the search and filter controls above the table, and an expanded variant comparison.)


A/B Testing Experiments Dashboard & Reporting FAQ

Why is my Conversions number lower than my Shopify order count?

Your Conversions number is lower than your Shopify order count because Rebuy counts only the orders where its experiment tag was present at checkout, which is a subset of all your store orders. Shopify counts every order your store receives, while the Conversions column counts only orders attributed to shoppers who were shown that variant. The two numbers are measuring different things, so they are not expected to match. Compare Conversions across the variants in your experiment rather than against your Shopify dashboard.

What is the difference between Total Revenue and Rebuy Revenue in an experiment?

The difference between Total Revenue and Rebuy Revenue is what each column counts. Total Revenue is the full store revenue from orders attributed to a variant, so it reflects the variant's overall business impact. Rebuy Revenue is only the portion of that revenue attributed to Rebuy, so it is always a subset of Total Revenue. Use Total Revenue to gauge the variant's total effect on your store and Rebuy Revenue to isolate the revenue Rebuy specifically drove. Rebuy Revenue is also the value divided by Conversions to produce the Rebuy AOV column.

Why does Rebuy AOV differ from my Shopify store AOV?

Your Rebuy AOV differs from your Shopify store AOV because it is calculated only from Rebuy-tracked orders, not from every order your store receives. Rebuy AOV divides the Rebuy Revenue for a variant by that variant's Conversions count, so it reflects the average value of the scoped subset of orders Rebuy tracked. Your Shopify store AOV includes all orders across your store. Use Rebuy AOV to compare variants against each other within an experiment, not as a measure of your overall store performance.

Did this answer your question?