One of the enduring themes on this blog is that none of the issues we face are truly new. Today, the old issue rearing its ugly head once more is data — specifically your data, specifically your data getting hoovered up to power modern Big Tech.
I think often about how the presentist AI discourse ignores the nearly identical arguments made about “big data” + “smart cities” a decade+ ago. As folks were celebrating the Knicks broadcast on Links, it struck me how their Google surveillance guts have been normalized.
These kiosks were conceived during the age of techno-optimism, which many in the industry confuse for a time when design was good. But all of the problems facing tech workers today were already baked in by 2016. The attitude tech brought to urban planning — elbowing experts out of its way to proclaim that Computer Science will solve all these trivial problems — is timeless.
Unfortunately, a city is not a computer, and so here we are a decade later, a few years into the “techlash” that gets blamed on any number of things instead of the real reason: none of the problems tech promised to solve ever got solved. What it did instead was unleash new problems onto urban populations. Primarily surveillance, where impacts can range from privacy violations by rich individuals to getting dragged into a legal nightmare by overhyped facial recognition technology. The societal panopticon continues to expand, driven primarily by fear-based narratives seeded by the industry itself:
To get people adopting the product, they explain, it was necessary to first create a crisis narrative, then use that narrative to cast biometric scanning of the product's users as an act of pre-emptive safety. In this way, World's AI-driven panopticon surveillance is portrayed as both natural and inevitable. All the while, the surveillance provider cashes in.
Another trend that seems to be back from the “golden age of UX design” is pointless wearable tech (hi, Sir Jony!). Ever since the dawn of the Apple Watch and its desperate search for a use case, the only thing tech has been able to find was fitness. Unfortunately, many have since gone on to confuse step counts with medically meaningful data:
People certainly often find it interesting to know a lot of parameters like their average heart rate, heart rate variability, etc. — but it is incredibly rare that it informs clinical decision-making.
Mostly, the impact of this tech has been to encourage quantification fixation. People have come to trust the number they see on a screen because it’s made to look like a fact — even though in reality, it is a rough estimate. Sort of how facial recognition tech that could at best promise “this is sort of like that guy” was dressed up to shout “this is definitely the guy!”
The new wearable form factor is “smart” glasses. The idea is from the same era as the smart city stuff (Google Glass in 2012, Snap Spectacles in 2016, etc) and floundered for the same reason: it didn’t actually have any purpose. But now they are back in a big way. Meta (and more recently, Snap again) have partnered with fashion houses to put out glasses that look merely regular stupid instead of “MIT in the 1980s” stupid, with billions in advertisements backing these things everywhere from TikTok to optometrist offices.
But between sending all their video recordings to contractors and making everyone users look at a victim of that shoddy facial recognition tech from earlier in the article, Meta’s glasses have already earned their reputation as pervert glasses.
At the same time, former Millennial darling Pokemon Go was caught selling the environmental data it recorded through players’ phones to military drone operators, and Anthropic is now sending your ID to Persona (the same Peter Thiel-funded startup that was too stinky for Discord).
All of this stuff is a preamble to a much more immediate concern for most of the people reading this: AI “note taker” apps that many tech workers now bring to meetings as a matter of course (or worse, send the app to a meeting in their place). The many issues with doing this got brought to light at an AI note taking etiquette event last month — but in the context of the rest of this newsletter, I’d like to add one more.
AI agents are not your agents. They do not represent your interests as a real (legal) agent would. And you have no idea what is now happening with those notes it took.
— Pavel at the Product Picnic