Your Conversion Rate Dropped. Probably Nothing Happened.

Why small numbers move on their own

Conversion is a series of independent events with a low probability each. When you have few of them, the count varies substantially from period to period even when absolutely nothing about the page, the traffic or the offer has changed.

Concretely: 400 visitors, 9 conversions, 2.25%. Next week, 400 visitors, 7 conversions, 1.75%. That reads as a 22% collapse in performance. It is two conversions. Two people had a different Tuesday.

The percentage is what does the damage. Expressing a difference of two events as a percentage change makes it look like a trend, and trends demand explanations. Somebody produces one — a Google update, a competitor, the new hero image — and the explanation gets acted upon.

Two conversions is not a trend. It is two conversions with a percentage sign attached to make it feel like one.

How much data is enough

Rules of thumb, and they are honestly rules of thumb rather than statistics:

• Fewer than 30 conversions: the rate is barely informative. Report the count, not the percentage.
• Around 100 conversions: the rate is reasonably trustworthy as a description of where you are.
• Comparing two rates: you need considerably more — typically hundreds of conversions per variant to detect the size of difference most page changes actually produce.
• Small effects need large samples. A change from 2% to 4% is detectable relatively quickly. A change from 2% to 2.3% — which would still be worth having — needs far more data than most sites generate in a quarter.

That last point is the one that undoes most small-business A/B testing. The test is not wrong; there is simply not enough traffic for it to ever conclude, and stopping it early because it “looks like” a winner is the same as not testing at all, with extra steps.

What bad sampling actually costs

The wrong number is not the expensive part. Four things follow from it:

• Time spent explaining noise. A fortnight of analysis on a fluctuation that had no cause.
• Good changes reverted. A genuine improvement that happened to have a quiet first week gets rolled back before it could show anything.
• Bad changes adopted. The mirror image, and worse, because it gets repeated as a lesson.
• Credibility. A team that reacts to every wobble stops being believed when something real does move — the boy who cried variance.

What to do when you will never have enough

Most small businesses will not generate enough conversions to run a statistically clean test, ever. That is not a failure and it does not mean measurement is pointless — it means the method has to change.

1. Aggregate over longer periods

Stop looking weekly. A month has four times the data and a quarter has twelve. Nothing about your page changes on a weekly cadence anyway, so the weekly view is pure noise generation.

2. Make big changes, not small ones

Testing two button colours needs enormous samples because the effect is tiny. Replacing a page that never states its offer with one that does can produce a difference large enough to see with modest data. If you cannot detect small effects, stop spending your traffic on them.

3. Measure the steps, not just the outcome

You may have 12 conversions a month and 3,000 people reaching the contact section. That intermediate number is large enough to move meaningfully and it tells you where the leak is. Scroll depth, form starts, form abandons, button clicks — all more abundant than conversions and all diagnostic.

4. Use judgement openly

Some things do not need a test. A page with no call to action should get one. A form asking for nine fields should ask for four. A missing phone number should be added. Running an experiment to confirm a known best practice on a sample too small to detect it is theatre.

5. Watch the direction over quarters

Three months at 1.4%, then three at 2.1%, is a real signal even without a formal test. Slow and unfashionable, and it is what the data you actually have can support.

Reporting honestly on thin data

Four habits that keep a small-sample report useful:

• Show the counts next to the percentages. “1.8% (7 of 400)” reads completely differently from “1.8%, down from 2.25%”, and it is the same information.
• Do not report percentage changes on small counts. “Down 22%” on nine conversions is technically true and functionally misleading.
• Say when you do not know. “Too few conversions to tell” is a legitimate and respectable finding, and it protects you from being asked to explain nothing.
• Set a decision threshold in advance. “We will review this in eight weeks or at 100 conversions, whichever comes first” stops the weekly re-litigation entirely.

Why the sample size belongs next to the rate

A calculator that returns “2.25%” and nothing else has given you a number stripped of the one piece of context that determines whether to believe it.

The conversion rate calculator shows the sample size alongside the rate, plus how many visitors it takes on average to produce one conversion — a figure that makes small numbers feel small in a way percentages never do. “One conversion per 44 visitors” is much harder to over-read than “2.25%”.

The target table does the same job from the other direction: it tells you how many extra conversions each target rate needs. When that number is three, you can see immediately that three is inside the range of ordinary weekly variation, and you will plan a longer measurement window rather than declaring victory on a good fortnight.

Free, no account, arithmetic in your browser, nothing stored. And once the number is trustworthy, the ROI calculator turns it into cost per acquisition and return — which is the number worth taking to a budget conversation, and worth being confident about before you do.

FAQ