A/B testing

Overview

A/B testing (also called split testing or, with more than two variants, A/B/n testing) is the standard method for making data-driven product decisions under uncertainty. Users are randomly assigned to a control group (the existing experience, "A") and one or more treatment groups (a variant, "B"), and a target metric — conversion rate, revenue per user, click-through rate — is measured for each group over the course of the experiment. Because assignment is random, any statistically significant difference in the metric between groups can be attributed to the change being tested, rather than to other confounding factors.

Key concepts

• Statistical significance: how confident we can be that an observed difference isn't due to random chance, usually expressed via a p-value or confidence interval.
• Sample size / power: experiments need enough users to reliably detect a real effect of a meaningful size; underpowered tests risk false negatives.
• Randomization unit: typically the user, but sometimes the session or another entity, chosen so that a single user consistently sees the same variant throughout the test.
• Novelty and time effects: results measured too early can be skewed by users' initial reaction to a change rather than its steady-state impact.

Analyzing A/B test results in SQL

Once event data is captured with a variant assignment per user, the analysis is often a straightforward aggregation:

Copy code
SELECT
    variant,
    count(DISTINCT user_id) AS users,
    count(DISTINCT user_id) FILTER (WHERE converted) AS conversions,
    count(DISTINCT user_id) FILTER (WHERE converted) * 1.0
        / count(DISTINCT user_id) AS conversion_rate
FROM experiment_events
GROUP BY ALL;

Running this kind of aggregation directly in SQL over raw event logs — rather than exporting to a separate statistics tool for every check — makes it easy to slice results by segment (device, geography, new vs. returning user) to check whether an effect is consistent or concentrated in a subgroup, which is often as important as the headline result.

Why it matters

A/B testing replaces opinion-driven product decisions with evidence about what actually changes user behavior, and is foundational to how most modern software companies iterate on products, pricing, and marketing.