IM Engineer Study Notes
← Back to list
AI & Data
#A/B테스트#온라인실험#무작위배정#검정력#실험플랫폼
Last updated · 2026-10-17

A/B Testing and Online Controlled Experiments

1. Overview

Definition: A/B testing is an online controlled experiment (OCE) technique that randomly splits users into two or more groups (control A / variant B), exposes them to different treatments over the same period, and tests whether the difference in a predefined success metric is statistically significant, thereby estimating the causal effect of the change.

When building a digital product, questions arise endlessly: "Will changing the button color raise conversion?", "Will swapping the recommendation algorithm increase dwell time?", "Will a new pricing policy grow revenue?" In the past such decisions were made by the opinion of the Highest Paid Person in the room (HiPPO) or by intuition, but intuition is often wrong. In fact, tallies from mature experimentation organizations such as Microsoft and Google show that only about one third of ideas teams believe are improvements actually move the key metric significantly; another third have no effect or even cause harm. This sobering reality is the starting point of a data-driven decision culture: "don't trust, measure."

A/B testing is powerful because randomization controls for confounders, letting us estimate a causal effect rather than mere correlation. Observational analysis only shows that "sales are higher on weekends"; it cannot prove "this feature raised sales." But when millions of users are split by a coin flip into two groups, the groups become on average identical in age, region, device, loyalty, and every other trait, so any observed metric difference is explained solely by the difference in treatment. This is why A/B testing is called "the clinical trial of product development."

Background and necessity

The statistical roots of A/B testing lie in R. Fisher's 1920s agricultural experimental design and the randomized controlled trial (RCT). Its explosive spread in the web era came because software can instantly deliver different versions to millions of users simultaneously and measure every action in logs. Experiments that took months offline now take days online, with hundreds running at once.

A second driver is the risk of a full rollout. Shipping a new feature to all users without validation can cause huge losses if the key metric drops. A/B testing is a safeguard that exposes a small fraction first to measure effect and side effects, and combined with feature-flag-based gradual exposure (discussed later) it becomes a means to learn while controlling risk.

Problem definition from a professional engineer's view

In an essay answer, describing A/B testing merely as "comparing two versions" fails. The core is to develop it along four axes: ① why randomization enables causal inference, ② how to quantitatively design the interplay of sample size, power, significance level, and MDE, ③ how to defend against reliability threats such as peeking, multiple comparisons, SRM, and network interference, and ④ how to institutionalize the experimentation platform and culture at the organizational level.

2. Overall Structure and Components

A/B testing is not a mere statistical calculation but a learning loop running from hypothesis to decision. The diagram below shows the overall flow and key components.

flowchart TD
  H["Hypothesis (change → expected metric lift)"] --> D["Design (OEC, MDE, sample size)"]
  D --> R["Randomized assignment (hash-based bucketing)"]
  R --> A["Control A (baseline)"]
  R --> B["Treatment B (variant)"]
  A --> L["Logging and metric aggregation"]
  B --> L
  L --> S["Statistical analysis (test, CI, SRM)"]
  S --> V{"Significant and trustworthy?"}
  V -->|"Yes: improvement"| SHIP["Full rollout"]
  V -->|"Yes: degradation"| STOP["Rollback / discard"]
  V -->|"Uncertain"| ITER["Redesign / extend / accumulate learning"]
  SHIP --> H
  STOP --> H
  ITER --> H

A. OEC — the Overall Evaluation Criterion

The metric that judges an experiment's success is the OEC (Overall Evaluation Criterion). The OEC must represent both short-term behavior and long-term value, and be hard to game. For example, if a news site makes "page views" its OEC, a dark pattern of splitting an article across many pages to drive clicks will be judged a "winner," while driving users away in the long run. Mature organizations therefore design user-value-centric OECs such as successful tasks per session or long-term retention per user.

The OEC may be a single metric or a weighted sum, but using too many metrics in the primary decision inflates the multiple-comparisons problem. In practice teams pick one primary metric as the decision criterion and set separate guardrail metrics—revenue, latency, error rate—that "must not degrade" to monitor side effects. If a recommendation change raised click-through but worsened page load latency, shipping is withheld even when the primary metric improved.

B. Randomized assignment and bucketing

Randomized assignment is the heart of A/B testing. In practice one concatenates the user id and the experiment id, hashes them (MD5, SHA, etc.), and maps the value into a bucket (e.g., 0–999), i.e., hash-based bucketing. This keeps the same user in the same group throughout the experiment (consistency) so the experience does not flicker, and guarantees independent randomization per experiment (orthogonality). Assignment must be per user; assigning per request causes the confusion of one user alternately seeing A and B.

The unit of assignment must match the unit of analysis. If you assign per user but aggregate a metric like "number of clicks" per session or event, the multiple observations of one user are not independent, so variance is underestimated and false positives rise. To prevent this, aggregate metrics per user or correct for cluster correlation with the delta method or bootstrap.

C. Treatment exposure and logging

By assignment, each group is exposed to a different UI, algorithm, or policy, and every exposure and action is recorded as event logs. A key pitfall here is consistency at the moment of exposure. If only the treatment group incurs extra computation and responds more slowly, the metric difference may be due to "the slowdown" rather than "the change itself." In fact, repeated experiments have confirmed that even a few hundred milliseconds of latency measurably reduce revenue and search usage, so the control group must carry the same measurement and exposure logic to remove measurement bias.

One must also define "was exposed" precisely. Results differ depending on whether only users who actually saw the feature are included, or everyone assigned. Like intention-to-treat (ITT) analysis in clinical trials, using everyone assigned reduces bias, but if many users never finished seeing the feature the effect is diluted. This triggering analysis design is an important decision that governs reliability and sensitivity.

D. Statistical analysis and decision

The null hypothesis (no difference between the groups) is tested on the aggregated metrics. Continuous metrics use the t-test; proportion metrics use the proportion (z) test or the chi-square test; and results are reported with the p-value and the confidence interval. A professional engineer's answer must go beyond the "p<0.05 means win" dichotomy and interpret the effect size and the width of the confidence interval together, distinguishing "statistical significance" from "practical significance."

For example, even if conversion rose from 5.0% to 5.1% with p=0.03, a 95% confidence interval as wide as [+0.3%, +4.1%] means "there is an effect, but its size is highly uncertain." Conversely, even when not statistically significant (p=0.2), a confidence interval of [−0.2%, +3.5%] carries the information that "harm is unlikely and the upside could be large." A decision must therefore weigh the direction, size, and uncertainty of the effect together with business costs and benefits, not a single p-value. Bayesian A/B testing presents results in a form directly usable for decisions, such as "92% probability B beats A, expected lift +0.4%," lowering the cost of communicating with business stakeholders.

3. Experiment Design and Reliability Assurance Process

A proper experiment fixes the sample size and duration before launch. The detailed diagram below shows the procedure from design to reliability verification.

flowchart LR
  subgraph PRE["Pre-design"]
    P1["Set baseline metric and MDE"] --> P2["Choose significance alpha and power 1-beta"]
    P2 --> P3["Compute sample size and duration"]
  end
  subgraph RUN["Execution"]
    P3 --> R1["A/A test to validate instrumentation"]
    R1 --> R2["Run experiment, monitor daily"]
  end
  subgraph VALID["Reliability checks"]
    R2 --> V1["SRM check (assignment ratio match)"]
    V1 --> V2["Guardrail and segment analysis"]
    V2 --> V3["Peeking defense (fixed horizon / sequential)"]
  end
  V3 --> DEC["Decision, documentation, knowledge accumulation"]

The quantitative relationship of sample size and power

Sample size is driven by four interacting factors: the significance level alpha (tolerated false-positive rate, typically 0.05), the power 1−beta (probability of detecting a true effect, typically 0.8), the variance of the baseline metric, and the minimum detectable effect (MDE) one wishes to detect. Intuitively, detecting a smaller effect with more certainty requires exponentially more samples. Approximately, the required per-group sample is inversely proportional to the square of the MDE, so halving the detection target from 2% to 1% requires about 4× the samples.

For example, to detect a relative 5% improvement (5.00%→5.25%) in a service with a 5% baseline conversion rate at alpha=0.05 and power 0.8 requires roughly hundreds of thousands of users per group. If daily traffic is 100,000, that means one to two weeks to a meaningful conclusion. Without doing this calculation before launch, you run with insufficient traffic and produce a power-deficient result—"no power to detect the effect"—which is then misread as "no difference."

A/A tests and SRM

To verify whether the experiment infrastructure itself can be trusted, run an A/A test (identical treatment to both groups) first. Repeated significant differences in an A/A signal bugs in the logging, assignment, or analysis pipeline. During execution, always check for Sample Ratio Mismatch (SRM). If a 50:50 design actually aggregates as 50.2:49.8 and deviates significantly under a chi-square test, the data has been corrupted by bot filtering, redirects, missed assignment, and the like, and the results cannot be trusted. Twyman's law—"any figure that looks interesting or odd is usually wrong"—reminds us why this check matters.

The peeking problem and multiple comparisons

A fixed-sample test is designed to look at the p-value exactly once, after reaching the pre-set sample size. Yet practitioners leave the experiment running, peek at results daily, and want to stop the moment p<0.05 appears. This peeking pushes the false-positive rate from a nominal 5% to 20–30%. There are two defenses. First, keep a fixed horizon and fixed sample and do not judge midway. Second, adopt sequential testing (e.g., always-valid p-values based on mSPRT) from the outset so that significance is guaranteed whenever you look. Testing many metrics or groups at once also inflates false positives through multiple comparisons, so apply Bonferroni correction or FDR (false discovery rate) control.

4. Comparison: Fixed-Horizon A/B Testing vs. Multi-Armed Bandit

A/B testing is not always best. The alternative multi-armed bandit gradually assigns more traffic to the better-performing variant during the experiment (explore–exploit balance, e.g., Thompson sampling), reducing the opportunity cost (regret) of the experiment period. The table below organizes the reasons for the differences and their practical implications.

Comparison axis Fixed-horizon A/B test Multi-armed bandit
Objective Accurate causal effect estimation / learning Maximize cumulative reward during the experiment
Traffic allocation Fixed (e.g., 50:50) Dynamic, shifting by performance
Fit Long-term decisions / judging across many metrics Short-lived content (headlines, banners) optimization
Statistical reading Clear effect size and CI Estimable, but needs bias correction

The essence of the difference is "is the objective learning, or revenue?" A decision whose value evaporates after a day, like a news headline, is better handled by a bandit to minimize loss. Conversely, high-value decisions whose outcomes will apply for months—pricing policy, a core recommendation algorithm—suit the fixed-horizon A/B, since you need accurate effect size and per-segment impact. Netflix using bandit-family methods for artwork (thumbnail) personalization while applying rigorous A/B to core recommendation changes is a real example of this distinction.

5. Practical Cases and Pitfalls

The experimentation culture of large organizations shows up in numbers. Booking.com is known to run more than 1,000 concurrent experiments, and Microsoft, Google, LinkedIn, and Airbnb each run thousands to tens of thousands per year. Google's anecdote of experimenting with "41 shades of blue" for link color is often cited as a symbol of extreme data-driven culture. There are also cases where a small change yielded large revenue. At Microsoft Bing, a seemingly trivial change to how ad titles were displayed, validated through an experiment, led to tens of millions of dollars in annual revenue growth—teaching the lesson that "even a small idea, when measured, can reveal great value."

There are also many pitfalls. Segment analysis warrants caution. Even when there is no effect overall, it may be strongly positive only for new users or degrade only on certain devices or countries, so examine per-segment impact alongside the OEC. But slicing segments endlessly after the fact mass-produces false positives through multiple comparisons, and illusions like Simpson's paradox, where per-segment direction flips against the overall direction, arise. So pre-register segment hypotheses where possible, and treat effects found in post-hoc exploration not as "conclusions" but only as "hypotheses for the next experiment."

There are also many pitfalls. The novelty effect is the phenomenon where a new feature lifts metrics only early on out of curiosity and fades over time, and the primacy effect is where existing users take time to get used to the change so early metrics come out low. Both are mitigated only by securing a sufficient experiment period (at least one full cycle, including weekends and weekdays). Network interference also matters. In services where users affect one another—social, delivery, ride-hailing—individual randomization violates the SUTVA assumption, so a cluster-randomization / switchback design that groups assignment by region or time window is needed.

6. Deep Dive: Variance Reduction (CUPED) and Institutionalizing the Platform

The two pillars of modern practice are statistical efficiency and platformization. The flagship of the former is CUPED (Controlled-experiment Using Pre-Existing Data). Using a user's pre-experiment behavior (covariates) to reduce the metric's variance lets you achieve the same power with fewer samples and a shorter period. Cases reducing variance by 30–50% are reported, which in traffic-scarce services multiplies experiment throughput several-fold. Choosing covariates while accounting for regression adjustment, stratification, and multicollinearity is a key design point.

The latter means institutionalizing experimentation as an organizational capability. Mature organizations integrate hypothesis registration → power calculation → automated assignment → real-time SRM and guardrail dashboards → result archiving into a single platform. This ① coordinates duplicate and conflicting experiments, ② reuses knowledge from past experiments, and ③ codifies guardrails so that non-experts can experiment with correct statistics. The "Trustworthy Online Controlled Experiments" framework established by Ron Kohavi and others is the standard reference in this field, offering a roadmap that raises capability step by step through the crawl–walk–run–fly maturity stages of an experimentation organization.

In the generative-AI era, experiment targets are expanding to LLM prompts, model versions, and RAG configurations. But because LLM output is non-deterministic and quality assessment is subjective, a hybrid experiment design that combines traditional A/B with LLM-as-a-Judge and offline evaluation is emerging.

7. Considerations and Implications

  • Application strategy — grow culture and platform together: adopting tools alone does not make an experimentation organization. Leadership's embrace of "don't trust, measure," the HiPPO's deference, and an evaluation system that rewards even failed experiments as learning must all advance together. Small organizations can start with an external experimentation SaaS, then transition to a self-built platform that internalizes SRM, guardrails, and power calculation as traffic grows—a realistic staged roadmap.
  • Trade-off — speed vs. reliability: shortening the experiment period allows faster decisions but lowers power and is vulnerable to novelty and day-of-week effects. Conversely, too long increases opportunity cost and contamination (interference with other changes). Ease this tension with CUPED and sequential testing, but the core is the balance of which metric to make primary and how far to set guardrails.
  • Ethics and regulation — users are subjects: A/B testing treats users as experimental subjects, so price discrimination, dark patterns targeting the vulnerable, and emotion-manipulation experiments raise serious ethical and legal problems. The 2014 large-scale emotional-contagion controversy is a representative warning. Place an ethics review gate in experiment governance from the standpoint of privacy law, consent, and fairness.
  • Outlook and related technologies — coupling with causal inference, gradual deployment, and MLOps: A/B testing is converging with feature-flag-based gradual deployment, canary releases, and the model-promotion gate of MLOps toward "deployment = experiment." Quasi-experiments (difference-in-differences, regression discontinuity) and causal-inference methods serve as complements where experiments are impossible, and the organization's data governance and metric standardization (data contracts) maturity becomes the precondition for experiment reliability.
  • Meta-analysis and knowledge as an asset: beyond the win/loss of individual experiments, meta-analyzing hundreds or thousands of accumulated experiments yields higher-level insight such as "what type of change tends to have the largest effect on average" and "is our organization's false-positive rate appropriate." Managing experiment results as searchable, reusable knowledge assets is the decisive difference between a mature experimentation organization and one that is not.

References

In one line: A/B testing is an online controlled experiment that randomly assigns users to estimate the causal effect of a change; it quantitatively designs OEC, MDE, power, and significance, defends against reliability threats such as SRM, peeking, multiple comparisons, and network interference, and through multi-armed bandits, CUPED, and the institutionalization of an experimentation platform and culture, it is the core instrument of data-driven decision-making that improves products by evidence rather than intuition.