Ask five marketers how long to run an A/B test and you will get five answers: two weeks, seven days, “until it hits significance”, one business cycle, or “until the tool says so”. None of those are wrong, but none of them are complete either. Test duration is not a rule of thumb you copy from a blog post. It is the output of a small calculation based on three numbers you already have.
This guide gives you the calculation, a worked manual example you can do on paper, and honest thresholds for low-traffic sites. The goal: stop calling tests on day three, and stop running tests that mathematically cannot ever produce an answer.
The short answer
Run your A/B test until you reach your pre-calculated sample size AND you have covered at least one full business cycle (usually 7 days), with a practical floor of 2 weeks and a practical ceiling of 4 to 6 weeks.
Unpacked, that means four conditions must all be true before you call a winner:
- You hit the sample size you calculated before launching the test.
- The test ran for a whole number of weeks (so weekdays and weekends are balanced).
- At least 2 calendar weeks have passed, even if the numbers arrived faster.
- You have not been peeking at the p-value and planning to stop the moment it dips below 0.05.
The two-week advice you see everywhere is a proxy for conditions 2 and 3. It is useful, but it says nothing about condition 1, which is where most tests actually fail. This explainer is clearer than most.
The three inputs that decide your test duration
Duration is not a setting. It is a consequence. Four variables drive it:
1. Baseline conversion rate
The current conversion rate of the page or flow you are testing, measured on the metric that matters (purchase, signup, lead, add to cart). Pull the last 30 to 60 days. If your baseline swings wildly week to week, that instability alone will eat your test.
2. Minimum detectable effect (MDE)
The smallest improvement you care about finding. This is the input marketers get wrong most often. If you say “I want to detect any lift, even 1%”, you have just committed to a test that needs roughly 100 times more traffic than a 10% lift test. MDE is a business decision: how big does the win have to be to justify shipping and maintaining the change?
3. Traffic volume to the tested page
Not sitewide sessions. The number of unique visitors who will actually enter the experiment each day or week, split across your variants.
4. Statistical settings
Standard defaults: 95% confidence (alpha 0.05) and 80% power. Power is the forgotten one. It is the probability of detecting a real effect that exists. At 80% power you miss one real winner in five, which is why underpowered tests keep producing “no difference” verdicts on changes that actually worked.
The manual calculation (no calculator needed)
For a standard two-variant test at 95% confidence and 80% power, sample size per variant is approximately:
n per variant = 16 × p × (1 – p) ÷ d²
where p = baseline conversion rate (as a decimal) and d = absolute MDE (as a decimal)
The 16 comes from 2 × (1.96 + 0.84)², rounded. That is all the statistics you need to memorise.
Worked example: a 3% baseline and a 20% target lift
Say your checkout page converts at 3% and you want to detect a 20% relative lift (3% to 3.6%).
- p = 0.03, so p × (1 – p) = 0.03 × 0.97 = 0.0291
- Absolute MDE: 3.6% – 3% = 0.6 percentage points, so d = 0.006
- d² = 0.006 × 0.006 = 0.000036
- n = 16 × 0.0291 ÷ 0.000036 = 0.4656 ÷ 0.000036 = 12,933 visitors per variant
- Total for A and B = 25,866 visitors
Now divide by traffic:
| Visitors per day entering the test | Days needed (25,866 total) | Verdict |
|---|---|---|
| 3,000 | 9 days (round to 14) | Comfortable |
| 1,000 | 26 days (round to 28) | Workable |
| 400 | 65 days | Too long, raise the MDE |
| 150 | 172 days | Do not run this test |
Always round up to a whole number of weeks. A 9-day test that starts on a Tuesday double-counts two weekdays and skews your data.
Why a smaller MDE destroys your timeline
Sample size scales with the square of the effect you are chasing. Halving your MDE quadruples the traffic you need. Same 3% baseline:
| Relative lift you want to detect | Absolute MDE | Visitors per variant | Total visitors |
|---|---|---|---|
| 50% | 1.5 pp | 2,069 | 4,138 |
| 30% | 0.9 pp | 5,748 | 11,496 |
| 20% | 0.6 pp | 12,933 | 25,866 |
| 15% | 0.45 pp | 22,993 | 45,986 |
| 10% | 0.3 pp | 51,733 | 103,466 |
| 5% | 0.15 pp | 206,933 | 413,866 |
The practical takeaway: on modest traffic, only test changes bold enough to move the needle by double digits. Button colour tests do not qualify.
Small-traffic sites: concrete thresholds
If you cap experiments at 4 weeks (sensible, because longer tests suffer cookie deletion, seasonality drift and stakeholder fatigue), here is what your traffic realistically allows at a 3% baseline:
| Weekly visitors in the test | Total over 4 weeks | Smallest lift you can reliably detect | What to do |
|---|---|---|---|
| 1,000 | 4,000 | ~50% | Radical redesigns only, or skip A/B testing |
| 5,000 | 20,000 | ~23% | Big layout, offer or pricing changes |
| 20,000 | 80,000 | ~11% | Standard CRO programme works |
| 100,000 | 400,000 | ~5% | Fine-grained iteration is viable |
A simpler sanity check: count conversions, not visitors
A quick heuristic that works across baselines: you want roughly 250 to 400 conversions per variant for a mid-sized effect. Under 100 conversions per variant, whatever your tool reports is noise wearing a suit.
If the math says “impossible”, do this instead
- Test higher in the funnel. Click-through on a hero CTA converts at 8 to 20%, not 3%. Higher baselines need far less traffic. Just confirm the micro-conversion correlates with revenue before you trust it.
- Test bigger swings. Compare two genuinely different page concepts rather than two headlines.
- Pool pages. Run the same change across all product pages or all landing pages instead of one URL.
- Accept 90% confidence and 80% power for low-risk changes. It cuts sample size meaningfully. Document that you lowered the bar.
- Use qualitative methods. Session recordings, five-user usability tests, exit surveys and heatmaps beat an underpowered A/B test for a site with 800 visits a month.
- Ship on judgement. If a change is obviously better for users and cheap to build, ship it. Not every decision needs an experiment.
Why stopping early is the most expensive mistake in CRO
Here is the mechanism. A p-value fluctuates as data accumulates. If you check it every day and stop the first time it crosses 0.05, you are not running a 5% false positive test any more. Repeated peeking with fixed-horizon statistics can push your real false positive rate toward 20 to 30%. That is how teams end up with a spreadsheet of 15 “wins” and flat revenue.
Two other early-stopping traps:
Novelty and primacy effects
Returning visitors react to change, not to quality. A new design can spike for 3 to 5 days simply because it is unfamiliar, then settle back. Running through at least one full business cycle lets that wear off.
Day-of-week and cycle bias
B2B traffic behaves nothing like Sunday traffic. Ecommerce payday weeks differ from mid-month weeks. If your buying cycle is long (high-ticket B2B, considered purchases), one week is not a cycle. Look at your own lag from first visit to conversion and cover at least one full length of it, ideally two.
When early stopping IS allowed
- The variant is broken: layout collapse, tracking failure, a dead payment button. Kill it immediately.
- Serious harm: a large, consistent drop that is costing real money. Stopping a loser early is far less dangerous than stopping a winner early.
- You are using a sequential testing method or Bayesian always-valid approach that was explicitly designed for continuous monitoring. Several platforms now offer this. If yours does, peeking is legitimate. If it does not, it is not.
Should a test ever run longer than 6 weeks?
Rarely. Long tests decay in reliability:
- Cookie deletion and multi-device use cause visitors to be reassigned, contaminating both groups.
- Seasonality shifts underneath you. A test spanning October into December in 2026 straddles pre-peak and peak shopping behaviour, and the two periods are not comparable.
- Other teams ship changes, run campaigns, or change ad mix, which alters your traffic composition mid-flight.
If your calculation demands more than 6 weeks, treat that as a signal to change the test, not to extend it.
A/B test duration checklist: when to call it
- Pre-registered sample size reached in every variant, not just in total.
- Whole number of weeks completed, minimum 2 calendar weeks.
- At least one full conversion cycle covered (two if your cycle is long).
- Sample ratio check passed. A 50/50 split should land near 50/50. A consistent 53/47 signals a delivery or tracking bug and invalidates the result.
- No mid-test changes. If you edited the variant, changed traffic allocation or launched a big campaign, restart the clock.
- Result is stable over the final few days rather than freshly crossing the threshold.
- The confidence interval is actionable. A lift of “+4% with an interval of -6% to +14%” is not a win, whatever the headline says.
- Segments make sense. Check mobile and desktop, new and returning. Use segments to understand the result, not to hunt for a winner in one slice after the overall test flopped.
The one-line rule to write on the whiteboard
Decide the duration before launch, write it down, and do not look at the results until that date arrives. Everything else in this article is a refinement of that single discipline.
FAQ
How long should an A/B test run, in plain numbers?
For most sites: 2 to 4 weeks. Two weeks is the practical minimum, four weeks is the typical sweet spot, and beyond six weeks the data quality starts to degrade. The exact figure comes from your sample size calculation divided by daily traffic, rounded up to whole weeks. There is more on it in How Long Should You Run an A/B Test.
What is the recommended sample size for an A/B test?
There is no universal number. Use n = 16 × p × (1 – p) ÷ d² per variant, where p is your baseline conversion rate and d is the absolute lift you want to detect. As a rough guide, aim for at least 250 to 400 conversions per variant, and treat anything under 100 conversions per variant as inconclusive.
Can I stop an A/B test as soon as it reaches 95% significance?
No, unless your tool uses sequential or always-valid statistics. With standard fixed-horizon tests, stopping at the first significant moment can raise your false positive rate to 20% or more. Wait for the planned sample size and duration.
How long should an A/B test on email run?
Email is different because the audience is finite and opens are front-loaded. Most of the response arrives in the first 24 to 48 hours, so a subject line test on a large list can be called in 4 to 24 hours. For click-to-conversion tests, wait 3 to 7 days, and make sure each variant went to a large enough random slice of the list (often several thousand recipients per variant for small differences).
My site only gets 1,000 visitors a month. Can I A/B test at all?
Not usefully for small changes. At that volume you would need an enormous lift to detect anything within a reasonable timeframe. Invest in qualitative research, heatmaps, user testing and obvious best-practice fixes, and focus A/B testing on high-traffic pages once you have volume. substack.com goes into the numbers.
Does A/B testing really work?
Yes, when it is powered correctly and run to completion. Most disappointment with A/B testing traces back to three fixable causes: underpowered tests, peeking and early stopping, and testing changes too small to matter. Fix the duration discipline and the method delivers.
Should I run the test for a whole number of weeks?
Yes. Ending a test mid-week overweights certain days of the week in your data. Start on a Monday and end on a Sunday, or any consistent 7-day boundary.
What about A/B/n tests with three or more variants?
Each additional variant needs its own full sample, and comparing multiple variants against one control inflates false positives. Budget roughly n visitors per arm, add a correction for multiple comparisons, and expect an A/B/C/D test to take about twice as long as a simple A/B on the same traffic.