A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem
Evals tell you whether the model can do the job. An A/B test tells you whether users care, and you need both before you ship. The things worth testing are the prompt wording, the model choice (GPT-4 vs GPT-3.5 vs OSS, weighing accuracy against cost and latency), and the generation parameters. Reported results include a chatbot reward-model variant with +70% conversation length and +30% retention, Nextdoor subject lines with +1% CTR after reward-function refinement, and Khan Academy's Khanmigo iterating on a latency-versus-math-accuracy trade. For tooling, Statsig (acquired by OpenAI for $1.1B in September 2025) bundles sequential testing, CUPED, and feature flags. GrowthBook is open-source and warehouse-native, with Bayesian, Frequentist, and Sequential engines, CUPED, SRM checks, and Benjamini-Hochberg and Bonferroni corrections. Pick based on how much you want to write warehouse SQL and whether "acquired by OpenAI" matters to your organization.
Learning Objectives
- Distinguish evals ("can the model do the job") from A/B tests ("do users care").
- Enumerate three testable axes (prompt, model, parameters) and pick the metric for each.
- Explain CUPED, sequential testing, and Benjamini-Hochberg multiple-comparison corrections.
- Pick Statsig or GrowthBook based on warehouse-SQL posture and corporate acquisition stance.
The Problem
You hand-tuned a system prompt and it feels better, so you ship it. Conversion changes by noise, and you blame the metric. Or you shipped a new model and conversion didn't move. Did the model degrade, or was the change too small to detect? You can't tell, because you shipped without an A/B.
Evals tell you whether the model can do a task on a labeled set. They say nothing about whether users prefer the output. Only a controlled online experiment answers that, and only if it has enough power, accounts for non-determinism, and corrects for multiple comparisons.
The Concept
Evals vs A/B tests
Evals are offline. You have a labeled set and a judge (a rubric, an LLM-as-judge, or a human), and the question is whether the output is correct, helpful, or safe on that fixed distribution.
An A/B test is online. Live users are randomized into variants, and the question is whether the new variant moves the user-level metric that matters.
You need both. Evals catch regressions before anyone is exposed, and the A/B confirms the product impact afterward.
What to test
- Prompt engineering: wording, system-prompt structure, examples. Metric: task success, user retention, cost/request.
- Model selection: GPT-4 vs GPT-3.5-Turbo vs Llama-OSS. Metric: accuracy on the task, cost/request, and latency P99, so it is multi-objective.
- Generation parameters: temperature, top-p, max_tokens. Metric: task-specific, usually output diversity against determinism.
CUPED (variance reduction)
CUPED stands for Controlled-experiments Using Pre-Experiment Data. You regress out the part of the post-period variance that the pre-period explains, then compare the adjusted values. Typical variance reduction is 30-70%, which raises your effective sample size without collecting any more users.
Both Statsig and GrowthBook implement it.
Sequential testing
Classical A/B testing assumes a fixed sample size, so peeking at the result early inflates the false-positive rate. Sequential tests control that rate under repeated looks. Always-valid procedures (mSPRT, Howard's confidence sequences) let you stop early when one variant is clearly winning.
Multiple-comparison corrections
Run 20 A/B tests at 95% confidence and you expect about one false positive by chance. Bonferroni tightens α per test. Benjamini-Hochberg controls the false-discovery rate instead, which is less conservative. GrowthBook implements both.
SRM (sample ratio mismatch)
An assignment hash randomizes users into variants. If a 50/50 split delivers 47/53, something upstream is broken, and the SRM check flags it. Both platforms implement it.
Statsig vs GrowthBook
Statsig:
- Acquired by OpenAI for $1.1B (September 2025). Hosted SaaS.
- Sequential testing, CUPED, held-out populations.
- Bundles feature flags, experimentation, and observability.
- Best fit: a team that wants one product and doesn't mind OpenAI ownership.
GrowthBook:
- Open-source (MIT) and warehouse-native, reading from Snowflake, BigQuery, or Redshift directly.
- Bayesian, Frequentist, and Sequential engines.
- CUPED, SRM, Bonferroni, and BH corrections.
- Self-host or managed cloud.
- Best fit: a warehouse-SQL shop where the data team owns the metric layer and wants OSS.
Non-determinism complicates power
The same prompt produces different outputs. Standard power calculations assume IID observations, and with LLM non-determinism your effective sample size ends up below the nominal one. Multiply the required sample size by roughly 1.3-1.5x as a safety margin.
Real case outcomes
- Chatbot reward model variant: +70% conversation length, +30% retention.
- Nextdoor subject lines: +1% CTR after reward-function refinement.
- Khan Academy Khanmigo: iterating on a latency-versus-math-accuracy trade.
The anti-pattern: shipping on vibes
Most senior engineers can name a feature that shipped because it "felt better," with no A/B behind it. Plenty of those regressed product metrics that nobody noticed for months. The A/B is what forces the question.
Numbers to remember
- Statsig acquired by OpenAI: $1.1B, September 2025.
- GrowthBook: open-source MIT, with Bayesian, Frequentist, and Sequential engines.
- CUPED variance reduction: 30-70%.
- LLM non-determinism: add a 30-50% sample-size buffer.
Use It
code/main.py simulates a sequential A/B test with fixed and sequential boundaries. It shows how the sequential bound lets you stop early when the effect is large.
Ship It
This lesson produces outputs/skill-ab-plan.md. Given a feature change, a workload, and a baseline, it picks the platform, the gates, and the sample size.
I'd run main.py before trusting any sample-size number in this lesson. It is stdlib Python, and it prints the fixed-sample size for a 3% baseline at 2%, 5%, and 10% lifts, then the same figure multiplied by 1.4 for the non-determinism buffer. The 1.4 is a constant in the script. It sits inside the 1.3-1.5x range above, but nothing in the code derives it.
The simulation is the part to read closely. It starts checking after 100 users per arm and stops when the z-statistic crosses sqrt(2 * log(1/0.05) + log(n)). On the 50% lift case it stops at 2,774 total users. On the 10% lift it runs to about 160,000 before stopping. On the 5% lift it never crosses the bound inside the 300,000-user cap and reports the stop as None. Early stopping pays off on large effects and gives you little on small ones, which is the trade you accept.
Two limits to keep in mind. The script has no CUPED code, so treat that section as concept only. And fixed_sample_size takes alpha and power arguments while hardcoding 1.96 and 0.84, so changing them does nothing. Exercise 1 is easiest to check if you fix that first.
Exercises
- Run
code/main.py. For an expected 5% lift with baseline 3% conversion, what sample size to 80% power? - Pick Statsig or GrowthBook for a healthcare-regulated on-prem customer.
- Design an A/B that tests GPT-4 vs GPT-3.5 on cost-per-resolved-ticket. What's the primary metric, guardrail metric, secondary?
- Your canary passes but A/B shows -1.2% conversion. Do you ship? Write the escalation criteria.
- Apply CUPED to a pre-period with 60% of the variance of post. Compute the effective-sample-size boost.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Eval | "offline test" | Labeled-set evaluation of model capability |
| A/B test | "experiment" | Live randomized comparison on users |
| CUPED | "variance reduction" | Pre-period regression to reduce variance |
| Sequential test | "peek-ok test" | Always-valid procedure allowing early stop |
| Multiple comparison | "the family error" | Running many tests inflates false positives |
| Bonferroni | "tight correction" | Divide α by number of tests |
| Benjamini-Hochberg | "BH FDR" | False-discovery-rate control, less conservative |
| SRM | "bad split" | Sample ratio mismatch; assignment bug |
| Statsig | "OpenAI owned" | Commercial all-in-one, acquired 2025 |
| GrowthBook | "the OSS one" | MIT warehouse-native platform |
| mSPRT | "sequential probability ratio test" | Classical sequential procedure |
Code
- main.py (python, 105 lines)