A/B testing
Design a controlled test before reading a conversion dashboard.
Quick learn
A fictional commerce app tests a clearer delivery estimate. Control has 400 buyers from 10,000 users. Treatment has 460 from 10,000.
Deep dive
Randomly assign eligible users and keep their experience stable. Choose the primary outcome, practical effect, minimum sample and duration before running. Check the expected allocation and monitor guardrails such as cancellations.
Worked decision
Purchase conversion increases from 4.0% to 4.6%, a 0.6 percentage point difference and a 15% relative increase. Compare uncertainty around the estimate and review guardrails before rollout.
When the decision changes
If cancellation rises enough to erase retained purchases, evaluate delivered orders or contribution per assigned user. If the allocation is unexpectedly imbalanced, investigate assignment or logging before interpreting the result.