Over the past two weeks, we talked a little about the high-level view of post-LLM design and how buyers view design now. Apparently “the nickd take on LLM” resonates, since all of my opinions are universally correct and many of you are wondering how to move going forward.
I’m going to write a little more today and a little next week, and then we’ll get back to our regular programming, because (despite what you may have heard) there’s quite a bit more to value-based design than how we respond to LLM. In the meantime, I might package some of my thoughts on post-LLM design into a smaller offering. Hit reply if you might be interested.
“Post-LLM value-based design” is a bit of a mouthful, and it’s also inaccurate, because value-based design is an evergreen principle that responds to conditions.
Value-based design still exists, even after LLM. I believe people will keep buying it. (Lord knows people keep buying our design: we’re on track for an all-timer of a year.)
To repeat, value-based design consists of three pillars:
- Researching customer behavior, desires, and values
- Measuring the impact of our work
- Experimenting with new ideas
All three of these can happen independently of LLM. Much of these should happen independently of LLM. (You don’t want LLM to take wild guesses on your KPIs.)
What LLM could do to the three pillars of value-based design
First, let’s talk about how LLM could affect the actual practice of value-based design.
Research
As I wrote last week, LLM is great for transcribing customer interview & usability test recordings. You’ll want to double-check & clean these up, though, since contemporary LLM often spits out recordings without line breaks, or with words repeated over & over again. The biggest tool going right now also fails to do attribution, so it’s sometimes unclear who is speaking.
LLM is considerably less great for batch research as of press time. One could, of course, dream of all sorts of science fiction ideas:
- Batch behavior recording analyses
- Heat & scroll map synthesis
- Screener questionnaires for recruitment at scale
- LLM-managed heuristic analyses (of which many startups currently exist!)
None of these are currently capable of outperforming value-based designers’ work. This is not an 80/20 question. They are underperforming by a long shot – if they exist at all. They could even provide material harm to a business.
One could argue that LLM is useful for synthesis of research, poring through lots of transcripts or behavior recordings. Even here, you still run the risk of inaccurate or too-surface findings. This could change someday, but it’s unlikely in the near future.
In short, research is a load-bearing strategic activity that shouldn’t be outsourced lightly. Transcription is relatively harmless because it’s an internal document and you’ve got multiple stakeholders checking the work. Now imagine an LLM parroting the same “best practices” that everybody has followed and adding a few incorrect ones for spice.
Measurement
You should not use LLM for measurement.
Experimentation
You should not use LLM to come up with experiment ideas. That’s what your research & synthesis processes are for.
LLM could someday be used as a way of amping up existing black-box Bayesian calculations for experimentation frameworks. I could see a world where you can get results faster. At the same time, for 13 years I’ve used actual statistics to call hundreds of experiments, because the dangers are too great when you avoid clarity. So I’m not too excited about that, either?
Will value-based design remain relevant?
I’m biased, but yes? This one feels extremely obvious to me?
LLM can make a value-based designer a tiny bit faster at their job, but it can’t replace any of value-based design’s load-bearing activities. In fact, I’d argue that value-based design is entirely rooted in clear human reasoning and real-world evidence, both of which are irrelevant to LLM.
To be clear, human reasoning & real-world evidence are not antagonized by LLM, nor are they replaceable by LLM. They exist separately from LLM.
LLM will also not render value-based design unprofitable by making some other technique more effective. Again, gathering, synthesizing, and acting on evidence is a fundamental technique of design. I truly cannot envision a world where they cease to be relevant – unless we suddenly start to disregard the needs of the customers we serve, which isn’t a world that I think any of us want to live in.
A lot of the discourse around LLM fails to recognize that a literal century of technological progress preceded it. Sure, you can ignore value-based design. But those who practice it will always profit.