The Conversion Rate Optimizer
SPECIFICATION
MKT-09
CATEGORYMarketing
FOCUSThe LIFT diagnostic model, the A/B testing "peeking problem" & Baymard's documented checkout-abandonment causes
BEST FORDiagnosing what's actually broken on a page before testing, and running tests that produce trustworthy results
METHODOLOGYWiderFunnel's LIFT model · Sequential-testing / peeking-problem statistics · Baymard checkout-usability research
FORMAT.md + .txt
COMPATIBLE MODELS
+ more
+ more
SKILL LEVELAdvanced
GENERAL ASSISTANT
We ran an A/B test on our checkout page for 5 days, checking results every morning, and it hit significance on day 4 so we shipped the winner. Was that okay?
Try A/B testing different versions of your page and see what converts better! Make sure your call-to-action stands out.
MKT-09 — THE CONVERSION RATE OPTIMIZER
We ran an A/B test on our checkout page for 5 days, checking results every morning, and it hit significance on day 4 so we shipped the winner. Was that okay?
That's actually a common and important issue - checking daily and stopping the moment it hit significance is exactly the 'peeking problem,' and it means the true false-positive rate on that result is higher than the 5% the significance threshold implies.
I wouldn't necessarily throw out the result, but I'd treat it as a hypothesis worth re-testing properly - commit to a sample size calculated upfront based on your baseline conversion rate and the effect size you actually care about, and don't check again until you hit it.
I wouldn't necessarily throw out the result, but I'd treat it as a hypothesis worth re-testing properly - commit to a sample size calculated upfront based on your baseline conversion rate and the effect size you actually care about, and don't check again until you hit it.
