🦆 Duck tests, why designers quit, usability test synthesis

I wrote a thread on Black Friday optimization, and learned that everybody loves it when I say objectively correct things. Noted!

A note on duck tests

I’m the latest guest on the Milk Bottle podcast, talking about your favorite topic: store design. In it, I talked a little about a major experiment that we ran for the Wander Club, where we provided tiered discounting for their entire product line. Conversions doubled overnight, and they ended up expanding their business significantly.

When I first sat down to call this test, I looked at the data, blinked, laughed, and thought “there’s no way that this can be that good.” So I checked for an outlier order, and found nothing. I laughed again, and decided I would wait a day to see if the data would stabilize.

The next day, I woke up, looked, saw the numbers had actually gone up in the past day, closed my laptop, walked across the street to a park, and stared at a pond with some ducks in it for an hour or so. Then I went back to my computer and wrote a report that said, in considerably more words, that we changed someone’s life.

We wrote about the test in our case study, of course, but not the ducks. I like calling these sorts of experiments duck tests now: the sort of results that rock your brain to such an extent that you have to stare at nature while recomposing yourself.

We don’t get duck tests very often. Maybe 7 have happened in the past 5 years? That’s a lovely number, but it won’t sustain an impactful design practice. Fortunately, the rest of our wins hover around 8% to 12% gains in average revenue per user, either through AOV bumps or shifts to conversion rate. That’s still not bad!

Would you rather have one swing-for-the-fences doubling in conversion rate, or six lower-risk tests that each bumped ARPU by 12% apiece? One option is far less likely to go viral in a LinkedIn post, the other option is far more likely to happen, but both result in the same increase.

If you’re thinking about hiring a value-based designer right now, I invite you to consider whether a brick-stacking practitioner would serve you better than someone who claims to promise the keys to the kingdom. What does your appetite for risk look like right now?

This week, for paid members

• Our weekly lesson covers a bit of evergreen content, on synthesizing usability tests.
• Our design of the week shows the best category navigation we’ve seen yet. How can you pull this off in your store?

Want in? Join us now – now named one of the best ecommerce communities going on the web.

Links & analysis

• An overview of why designers quit their jobs. Turns out value-based designers like value-based maturity. Who knew?
• Nobody ships design. They ship the built results of design. This article expands on this to make an interesting point about how one must design for collaboration within a team.
• Following up on last week’s beautiful & inspiring factual statement regarding Figma’s sudden irrelevance, this post reminds us all that design tools are fleeting, but the practice of design is evergreen. More from Casey Newton.

This week’s paid lesson: How do you synthesize a usability test?

This week’s lesson is for paid members. Sign into our community to read it, or join us today to get access.