Experimentation & A/B Testing
One test tells you what happened. A program tells you why — and what to try next.
Start a projectMost stores run tests the way they run redesigns — a hunch, a launch, and a guess about what moved the number. Without proper sizing, a variant that looks like a winner after four days is often just noise settling, and a real result gets called too early, or missed entirely because nobody was watching long enough to see it clearly. Either mistake leaves you making decisions on evidence you can’t actually trust.
We run experimentation as a program: a hypothesis backlog built from research, tests sized before they launch, and results called honestly — wins, losses, and the flat ones that teach you as much as either. AI covers more ground — more variant drafts, more session data reviewed than a small team could get through by hand — but it doesn’t size a test, read a result, or decide what ships; those calls stay with our people. What builds up over time is an evidence-based picture of why your customers buy, not a stack of one-off guesses.
Built and run, end to end.
A hypothesis backlog built from research
Ideas come from analytics, session recordings, heatmaps, and support tickets — not a brainstorm. We turn what the data actually shows into a prioritized backlog, so the next test runs because it's likely to matter, not because it's next on a template.
Tests sized to actually finish
Before anything ships we size the test: how much traffic it needs, how long that takes, and what a real result looks like versus noise. We don't peek early and call a trend a winner. A test that ends on schedule with a clean read beats one stopped the moment it looks good.
Honest calls, including the losses
Most tests don't win. We write up losses and flat results with the same care as wins, because a test read honestly still tells you something true about your customers even when the metric doesn't move. The record of what didn't work is part of what you're paying for.
Winners shipped into the live theme
A test that wins and never ships is wasted work. We build the winning variant into the production theme, retire the losing path, and move the backlog forward — so the program compounds instead of restarting from zero each time.
Questions, answered.
How much traffic do we need to run a testing program?
Enough monthly conversions for a test to reach statistical significance in a few weeks, not raw sessions. Lower-traffic stores can still test, but we lean on bigger swings and qualitative evidence — session recordings, heatmaps — rather than fine-grained variants that would take months to read honestly.
How long does a single test run?
As long as the sample-size math says it needs — typically two to six weeks, depending on your traffic and the size of the change. We calculate the duration before launch and don't call the result early just because a lead opens up.
What happens when a test loses?
We retire the variant, document what we learned, and move to the next hypothesis. A loss still tells you something real about how your customers decide — that's why it goes in the record instead of getting quietly deleted.
What testing tools do you use?
Whatever fits your platform and traffic level cleanly — native experimentation tools where the platform has them, or a dedicated testing tool where it doesn't. The tool matters less than the sizing discipline and the honesty of the read.