For those of you keeping track, this was the first time the Picnic has had a two-week hiatus. There were a bunch of reasons for it, chief among them being that it was the Labor Day holiday and I was in Canada taking photos of herons. But also, I didn’t really want to write the issue, because it’s about AI and I’m tired of talking about AI.
Ironically, this would have been the perfect time to spin up Chatty G (to borrow Candi Williams’ name for the thing) and ask it to extrude 800-1000 words based on my draft. Hell, if I did that, I could probably send out a Picnic every day instead of every week.
But what would be the point? Almost every other website is already written in that hauntingly uncanny voice that is instantly recognizable as AI.
It’s hard to knock an LLM off its default narrative persona, because that’s what the model’s user interface is. … we encounter op-eds, white papers, ad copy, erotica, propaganda etc with a crazy diversity of subjects and rhetorical purposes all spoken by the same poorly-defined implied narrator. We learn to recognize the subliminally unmoored feeling it gives us.
“Uncanny” is, in fact, the technically accurate term. Naomi Saphra goes into detail on how AI-generated text (I will never call it “writing”) has dragged all electronically mediated interfaces down into the uncanny valley. The companies who produce our software claim they are making things easier and more intuitive, and then go on to slap on a massive cognitive load onto every user: the subconscious work of figuring out if the service experience you are having is real or not.
The cognitive load of the scam economy
That hypervigilance is warranted: AI is best at scaling up scams, so now scams proliferate, often indistinguishable from ordinary publishers caving to the pressure of feeding the algorithm with “optimized” AI-generated slop. Everything we touch on the internet now has the dubious, sticky texture of a scam. Authors are being overwhelmed by waves of book club scams. Colleges are swamped with fake students farming financial aid. The same people who are funding frontier labs are pouring money into services that flood social media with spam. Job applicants are having their personal information stolen by entire phantom companies, via simulated Slack instances — and recruiters have to dig through piles of fake resumes that have never been easier for unscrupulous candidates to produce.
Human-to-human interactions are not safe, either. Even healthcare professionals (who should honestly know better) are hiding their AI use from patients, resulting in made-up referrals and incorrect prescriptions.
Over 50 years ago, Herb Simon described an attention economy — the world evolving from a system in which information was the bottleneck to one where an information surplus overwhelmed human capacity to process it. For Simon, the new bottleneck became the amount of things to which we could devote our attention. But Simon could not predict that some of that information would go from merely irrelevant (low value) to adversarial (negative value). In a scam economy, a constant, low-level anxiety pervades our daily lives: is the thing I’m engaging with real? Or is it a phantom, a facsimile, even an antagonist? Count the fingers, count the teeth.
I find it viscerally disgusting; a cold shudder like someone’s poured jelly down the back of my neck. I hate that it’s everywhere; I hate that when I read basically anything now I’m constantly on alert, twitching like a schizo in an underpass. Is this thing really what it says it is? Is this person actually a robot in disguise? Nice little personal essay you’ve got there, lady, but I know what you really are; time to get my knife out, time to start digging around under your skin until I find the wires.
Wang mentions a suite of AI detection tools, now rising in popularity as people rebel against being fooled. In Europe, disclosure of AI use is now a legal requirement. Some companies (Libby, for instance) are even kind enough to build those filters directly into the product for worldwide users.
Others fumble the ball completely, because their business model depends on slop pipelines; consider SurveyMonkey telling survey-takers that their submission is slop so LLM-powered bots can take a mulligan, or Grammarly laundering its own AI generated content as human. Social media sites, naturally, refuse to let you filter out AI posts, even if they are the ones providing users with tools for extruding post-like content, because they can shift more ad inventory this way.
Your design thinking has been incepted
At this point, it almost doesn’t matter whether the slop we read is AI-generated. That soulless, shiny plastic cadence comes from somewhere, after all: reams of training data, chiefly from placed like LinkedIn, that some hapless data scientist tagged as examples of good writing. Now that style is being fed back into the ecosystem, reinforcing a monoculture we all hate and yet feel obliged to participate in.
GenAI is different from previous technologies that merely spread information, the researchers say, in that it actively shapes it. Studies suggest that AI tools can flatten language use, creativity and cultural values, and there is evidence that the technology can even influence the decisions we make, the way we act and the opinions we hold.
And this is why I dragged you through 1000 words on the harms done by slop. These harms don’t only exist in an abstract “out there” that we engage with after 5pm on weekdays. They do also exist there, and make those hours far more tedious than they need to be. But the more you read the not-quite-human content, the more you internalize it. And the more it begins to eat away at your own writing, and your own thinking.
To get a little LinkedIn for a moment: consuming slop trains you to write slop, and so it directly undermines your value to your employer and your competitiveness in the job market. The bread and butter of software companies is knowledge work; if cognitive diversity is its only competitive advantage, why would they want a worker with a commoditized thought process?
Maintaining cognitive diversity is … a functional requirement for the kind of computation that collective intelligence depends on. … an organisational design practice that treats cognitive profile distribution as an analogue substrate to maintain, not individual variation to manage toward a norm … would recognise that anything enforcing cognitive convergence at scale — process environments, competency frameworks, uniform AI policy — is making a trade on behalf of the collective that the collective has not been asked to consent to, and may not be able to reverse.
One of the most common pieces of advice I give to people trying to express themselves in natural, human-sounding ways is to explain to me what it is they want to say, and then just write that down. This is not, however, advice that I often take myself. So it was refreshing to hear it directed at me for once, in a touching essay by Mike Monteiro about the importance of writing using your own brain. Even if it’s hard. In fact, because it’s hard.
This piece was going to be a lot longer, so tune in next week for part 2 about the extent to which reliance on LLMs replaces your thinking with someone else’s thinking (and, if it’s not already obvious, why that is a bad thing).
— Pavel at the Product Picnic