Howdy, picnickers!

It’s always difficult to strike the right balance between breaking the established tone for the sake of a holiday, and not acknowledging it at all. This is my attempt:

Back to your regularly scheduled newsletter. 

Reading Material

The word of the day is measureship, as in “leadership by measurement.” In an environment where “do whatever you did last year, but 20% harder” passes for strategy, managers are increasingly disengaged from an understanding of value that isn’t reflected on the default dashboard that came with their Jira install. I’ve ranted about this before but Paul’s post is a succinct summary of the problem.

Unfortunately, in the 15 years since the term vanity metrics (is this the term’s earliest appearance?) has been with us, managers forgot that what you choose to measure matters just as much as what that measurement says. The tyranny of Goodhart’s Law foretells that measureship will inevitably produce lousy results, and so it does. 

Everything except the topline metric suffers:

• “After the shift to an open-plan office space, the participants spent 73 per cent less time in face-to-face interactions, while their use of email and instant messenger shot up by 67 per cent and 75 per cent respectively."

• “A report prepared for the Pennsylvania Department of Education in 2019 found that compared with other schools, “cyber charter schools have a consistent negative effect across all outcomes except graduation.”

• “We find that private ambulances reduce costs and perform better on contracted measures such as response time, but perform worse on noncontracted measures such as mortality.”

This is the consequence of a decision structure that is not merely data-determined (managers looking to the data to tell them what to do) but data-subordinated (managers making decisions in order to create the data they want). This is similar to what Jonathan Korman calls decision-based evidence making but somehow manages to be even more craven.

For example, we now have popup ads on car dashboards because “ad impressions” is a metric that goes on the big dashboard management sees, and presumably “number of people we killed” is not.

Chuck Klosterman has a delightful book called “What If We’re Wrong” that examines what made those outstanding products of yester-year so memorable and enduring, and what didn’t. It turns out that managing to metrics was precisely the right way to condemn your work to obscurity. Who knew!

It’s also a really good way to kill all the enthusiasm in your team, but between layoffs, RTO, and kowtowing to authoritarian demands, it’s almost as if this is the point.

The Timeline

Measureship is the eternal enemy of good design; consider for example a “time on site” metric that makes every stakeholder feel warm and fuzzy inside for every extra second users can’t find what they are looking for. Over on LinkedIn, Joanna Weber has an excellent example of the kind of anti-design absurdity that measureship leads to.

Measureship undercuts another key power of design: the feedback loop of critique. When the decision-maker’s point of view on a product is “do whatever makes the number go up” they are not only tracking trailing indicators, but they are not capable of saying no. And the ability to say “no, because this does not take us where we want to go” is the key to effective product decisions. They can only say “I don’t like it” which is useless feedback.

We need to reach no further than Facebook to see what kind of design decisions come out of a product culture infiltrated by measureship:

Wendy Castleman has a good discussion going about how we might incentivize good design by picking better measurements. However, not all designers are framing the discussion around data in productive terms - this post comparing qualitative research to journalism blew up on my timeline the other day (it is worth reading the comments for some great defenses of both qualitative methods and journalism; mine is here).

Good Questions

“Make number go up” was a very common request from my AWS customers. Usually the number was something like revenue, share of wallet, conversion rate, or some other generic metric you can pull out of an entry-level business course.

But having decided on the number, they were stuck on how to proceed. Ironic, because between me and them, they had all the answers - how their industry works, how their company makes money, what their competitive advantages were.

But I had the questions. And the question I started with was:

Why isn’t the target audience already using the product? Why have previous roadmaps towards inevitable success failed?

We unpacked the challenges behind “make number go up” in the same way that in order for “make it easier” to be meaningful, you first have to unpack all of the things that make it hard.

However, it’s important to note that this “why” is not the first “why” of five. From this point on, “how” is far more valuable than “why.”

– Pavel at the Product Picnic