Spring is here in the Northeast, just like it was here two weeks ago, and three weeks before that. But this time is for real. Definitely.

A similar sentiment follows the design process, which continues to be declared dead in the hopes that someday the accusation will stick. I think that unfortunately this proclamation will never entirely go away — at least, not while designers continue to be confused about the shape of the design process, and our place in it.

The double diamond describes only one dimension of the process: it shows how an idea moves in time, from its inception to its execution. But if you could take a vertical slice of the diamond and look at its flat side, what you’d see is a set of two feedback loops. One between the team and the external customer, and the other with the internal customer (more on this idea and its implications here).

Once you understand this, the distinction Andy Polaine makes this week slides into focus: what everyone is suddenly obsessed with producing (via vibe coding) is not actually prototypes. It’s demos.

The only thing these demos are helping you test is whether your stakeholder likes what they see (the first loop) and as soon as they say “yes,” it becomes good enough to ship.

Whether that second loop (releases go out, measurements come in) ever gets tracked or not is not something I’d be willing to put money on. Because once the demo is productionized, it goes from the realm of delivery velocity (which gets you shoutouts and promotions) into the realm of maintenance (which tends to be ignored even as it eats up more than half of the team’s bandwidth).

Nothing ends up being learned, which is to say integrated into decision-making about the product. The impact might be reflected on some dashboard that everyone ignores — but until that change is incorporated into decision-making, it can’t actually be described as learning.

And the sooner designers reckon with how knowledge is actually created and acted upon in their organizations, the faster they can start making prototypes that actually help them learn.

It’s unfortunately very common in tech to reach for the ironclad armor of “data driven” without knowing anything about what drives data. But there are professionals who study that for a living, and now we have a study to confirm what the humanities have been saying for decades: quantitative data is always, always skewed by the extremely qualitative work of choosing what to measure and how to measure it.

There is a childish notion that being “data driven” allows one to avoid distractions such as office politics. But rather than allowing us to rise above that morass, data-gathering is politics. This should also be obvious: data is power, and the process of creating and distributing power is always political. Far from being a neutral arbiter of truth, data is the medium through which prevailing power is exercised:

And chasing this power is not a neutral activity, either. Attempting to shape the narrative through carefully selected metrics instantly runs the whole organization afoul of the old adage: when a measure becomes a target, it ceases to be a good measure. And if your target is an aggregate measure, it screws you twice: once through the questionable logic of the metric itself, and then a second time through the inevitable gamification exercises that spontaneously occur downstream of that metric.

Any experienced researcher will tell you that the plural of anecdote is not data, but when an anecdote conflicts with the data, you should start paying attention. This is because qualitative analysis is what lets us make sense of the numbers. It’s the only way to tell whether our metrics are measuring the wrong thing.

Good qualitative research is fundamentally different from good quantitative research. This should be obvious — after all, they are different things, different disciplines, different formats. And yet, quant dominates our imaginations to such an extent that qual ends up being judged by the same metrics. We expect qualitative research to get better with scale (it doesn’t), simply because its sibling, quantitative research, gets better with scale.

In fact, I’m on the record as going one step further: often, scale defeats qualitative research by drowning out the outliers — the voices that are not being heard — with a tidal wave of statistically average responses (it will not surprise you that jokesters are now using generative AI to do this exact thing).

To close the loop on the theme of this essay: this is exactly why AI-generated prototypes are not working, and have not helped anyone do anything ever. Some have accused me of going too far with this assertion, but I stand by it, because it is rooted in the very nature of what a prototype is (and is not), and what makes it successful (or does not). As above: a prototype is not a demo. It does not succeed when we have shown a thing to a stakeholder. Shoving out more prototypes is not a heuristic for success; it is a heuristic for failure because it shows that you don’t know what you are trying to learn.

One of the reasons I love this field is that it ties in ideas from every sphere of human endeavor. Among my favorite nuggets is “cognitive estrangement” which comes from science fiction literature. Simply put, it’s the idea that placing a familiar topic amid lizardmen and laser guns will cause a reader to think about it without any preconceptions. This has obvious applications to our field, where we often have to nudge our stakeholders out of the business-as-usual rut.

But this same technique can (must?) be applied to our own work. Because while UX design is mesmerized by the lure of the visual fidelity anglerfish, our sibling practice of content design has fared somewhat better. Because the analog of a design system is a content model, and its nature as a model means that conceptual integrity is a requirement rather than a nice-to-have.

If you have been making (generating?) prototype after prototype without any visible impact, try this next time: put the visuals to the side. Start with the content that your users need to understand the options available to them, and to help them form a meaningful response.

More on content-first prototyping next week!

— Pavel at the Product Picnic