Meet fuck split
As an optimizer, your job is to figure out the truth. It’s very human to get attached to your hypothesis or design treatment, and it can hurt when your best hypotheses end up not being significantly different. A year later the conversion rate of their site is the same as it was when they started. You should not call tests before you’ve reached 95% or higher.Check the test results across segments to see if significance was achieved in one segment or another (great insights lie always in the segments – but you also need enough sample size for each segment). We need to rule out seasonality and test for full weeks.In any case, you need to improve your hypothesis and run a new test. You achieve 98% confidence and 250 conversions per variation in 3 days. Run a conversions per day of the week report on your site, see how much fluctuation there is. Thursdays make 2x more money than Saturdays and Sundays, and the conversion rate on Thursdays is almost 2x better than on a Saturday.You NEED TO calculate the actual needed sample size ahead of time, using sample size calculators like this or other similar ones.
This is a pretty useful tool for understanding the relation between uplift percentages and needed sample sizes:
is an [in]famous long-running Animated Series by Trey Parker and Matt Stone on Comedy Central.