Color, metric strategies, minimum sample size
Go read this essential post on No Best Practices now. I think it’s insightful, it taught me a lot, and I have thoughts.
Ok, did you read it? Great.
I love this post, and it got me thinking about a lot of the things I do to determine whether a certain experiment – or experimentation at all – is a good idea. Yes, below a certain point, you should buy a teardown and run with it. Past a certain point, you should hire a value-based designer to run experiments.
From what I’ve seen in my work, the cutoffs between each of these are a lot fuzzier. For the purpose of a blog post, it probably makes the most amount of sense to say “if you make $X, do this,” but we can go deeper here. So I’m writing up a little bit of clarification here to show the art that goes behind a post like this, in the hopes that it may cultivate your own intuition around what to do and when.
Traffic volumes & time
I’m guessing the reason why these numbers have been selected, and why a lot of the specific tactics have been chosen, is because of the idea of a minimum sample size. This is the amount of traffic that you need for a specific experiment to run.
Go to this palace of Times New Roman, slam the “Statistical power 1−β” slider to 95%, and play with the numbers a bit, and you’ll see what I mean. Lower conversion rates require more traffic to a page. Higher conversion rates require less traffic.
Sample size divided by traffic gives you a certain amount of time that your experiment has to run. And really, you just want to know how long to run the test. Look at the traffic volume and conversion rate going to a specific page, and plot accordingly.
Your whole business has a conversion rate, but every page has its own conversion rate, too. You run experiments on pages, not on your whole business. Your cart is likely to “convert” much higher than your home page because you’ve already filtered all of the bounces from direct traffic and you’re working with a lot of wallet-out consumers. That’s presumably why Alex Greifeld from No Best Practices recommends that you run cart & checkout testing – because you can get to statistical significance faster.
But I want to offer another point here, which is that down-funnel pages get way less traffic than your home page. In practice, this means that while you’re likely to get a high sample size for your home page, you’re also about as likely to hit it.
Revenue goals
All revenue numbers presuppose a certain AOV, which then connects to transaction volume. Some industries will have low AOV and high interest; others will have high AOV and more window-shopping.
Alex is writing to a DTC audience that continues to rely heavily on paid ad spend, where AOV probably hovers in a relatively predictable range. But if you’re reading this and think your store might represent a special case, then I invite you to look beyond revenue numbers as to whether value-based design is worth your time.
Again, sample size is our north star. Look to your existing conversion rate to determine whether this makes sense. I’ve run lots of experiments for $7M businesses that are low-involvement, high-interest, and low-AOV. It worked. I’ve also tried to run experiments on stores that had a $1,000+ AOV and hardly any purchase volume. It was harder. In value-based design as in life, it depends.
Where to go
Finally, I think Alex might have it backwards with respect to the order of pages to test, at least for most of you reading this. Checkout tests are likely to yield results because they already operate off of significant customer interest, but checkout is locked down heavily on many platforms now and it’s hard to get durable experimentation results from that page. Furthermore, because of sample sizes, it’s not immediately clear to me what one page would do over another in a funnel. They all matter.
I tend to look more holistically at a store. What pages have seen the least love? What has seen the most design inertia? What are the most politically fraught in an organization? In practice, that for me has meant cart, followed by home, followed by product detail, for most of my clients. That might be different for you! But really, this is why you should hire a value-based designer to prioritize experiments for you instead of saying “do checkout.” Prioritization is done independent of page type.
Gosh I loved that post
I think this is the best summary of value-based design that I’ve seen for those who buy design that I didn’t write myself. I’m linking this and offering some clarifications for our audience because I think it will be of deeper interest. Take a look at Alex’s post and let me know what you think about what I’ve written here. I invite all forms of feedback!
This week, for paid members
- This week’s paid lesson teaches you how to find the right strategies based on the metrics you’re gathering – and how to gather the right metrics to fit your business’s strategy. A rare quantitative lesson from us, but it really speaks to the subtler sides of research!
- Our design of the week shows a really interesting size & color selector that works perfectly on mobile. What can we learn from it?
Want in? Join us now – named one of the best ecommerce communities going on the web.
Already a member? Log in here and take a look at what’s new.
Links
- Color meanings across cultures. I had no idea about some of these!
- More on color, this time for determining what colors to use and where in your design.
- Meta-analysis in experimentation, or testing about testing. After a certain level of sophistication, this activity becomes necessary. How do you do it well?
- On centering.
- What clients expect from value-based work. This is why I spent 12 years waffling on whether to write up a process. We care about our process. Do buyers care? They do not care.
- Ratings design guidelines, from Baymard.