Instagram buzzes your phone and re-displays a message you received several days ago that you haven’t replied to yet. When you unlock your phone and tap the notification, instead of taking you to the message directly, it takes you to your feed. At the top of it is something that grabs your attention, like a funny comic or a half naked person. You habitually scroll to grab another dopamine hit of content. The next post is an advertisement with an AI generated image that you find oddly compelling, and the one after that is a political influencer pointing out that that politician you hate did another awful thing.
We’re under psychic attack. The purpose of the assault is to commoditize our existence, so as to increase shareholder value. Corporations are profit-generating algorithms, and like any good algorithm with no termination criteria, they’ll go happily runaway. Don’t worry about paperclips, we already have artificial group intelligences optimizing the generation of profit.
Humans can’t be told what to do, we’re famously very stubborn. That’s fine, ask Claude “find a way to convince a population to make us money - don’t make mistakes” and it’ll happily loop into the solution that Instagram did: You’re more than willing to provide unlimited training data to Instagram, as well as eyeballs for its advertisers, in return for ragebait and FOMO pics that Instagram itself doesn’t even need to generate.
What’s this roaring waterfall of content doing to us, our behavior, our brains? Surely we wouldn’t be buying all the things we’re buying if we weren’t exposed to so many Instagram ads. If not, then Instagram’s been scamming its advertisers! All of this noise is making us participate in a fever pitch of consumerism. We’re not just competing with our neighbors to have the best lawn anymore, we’re competing with Los Angeles failsons thousands of miles away to have the best bottled water we use for our morning facewash. Our FOMO has been globalized.
Influence
I’m beginning to feel more and more lately that the most powerful thing a person can do is develop their rhetorical ability. By rhetoric, I mean, the behaviors, styles, words, and strategies people implement so as to influence, convince, convert, and generally modify the behavior of other humans.
Excellent rhetoricians have achieved the incredible: Charles Manson convinced a group of kids to murder nine people. Jim Jones convinced 900 people to kill themselves - yes, some at gunpoint, but he also convinced those holding the guns to also kill themselves. I think it’s fair to assume that figures like Jesus of Nazareth or Muhammad were skilled rhetoricians, and look what that skill can earn you: millennia-old religions.
What does it take to convince people to die en masse, or believe in new gods, or subject themselves to pain and torment for said new gods? What does it take to convince people to rise in revolution? To march thousands of miles in the snow? Is it a pattern of speech? A way of looking at people? A specific combination of words? Tone of voice? A style of dress, hair? A well-formed strategy of public appearances?
Neal Stephenson has written several times about the seemingly magical power of words, and what it means when machines start learning how to leverage that power. In “Snow Crash,” a virus is ravaging the internet. Its form is merely a bitmap that, when viewed, sends the viewer into a coma. It’s suggested that this is an ancient attack vector re-discovered, a Sumerian “Nam-Shub”:
“A nam-shub is a speech with magical force. The closest English equivalent would be ‘incantation,’ but this has a number of incorrect connotations.”
“Did the Sumerians believe in magic?”
The Librarian shakes his head minutely. “This is the kind of seemingly precise question that is in fact very profound, and that pieces of software, such as myself, are notoriously clumsy at. Allow me to quote from Kramer, Samuel Noah, and Maier, John R. Myths of Enki, the Crafty God. New York, Oxford: Oxford University Press, 1989: ‘Religion, magic, and medicine are so completely intertwined in Mesopotamia that separating them is frustrating and perhaps futile work…. [Sumerian incantations] demonstrate an intimate connection between the religious, the magical, and the esthetic so complete that any attempt to pull one away from the other will distort the whole.’
Later, in “Fall, or Dodge in Hell,” he explores the possibilities of this power if AI is applied to it:
“Do they ever cross the river?”
“When they get desperate or some meme convinces them it’s a good idea. They follow these weird edit streams. No one knows where they come from. I looked at one of them once. I thought it would be conspiracy-theory stuff but it wasn’t even coherent enough to be called that.”
“Hmm. Yeah, I guess the whole point of a conspiracy theory is to offer a kind of false coherence.”
“That’s right… but what I saw didn’t even rise to that standard—didn’t even know about it. It was—well, just plain weird. Algorithmically generated mishmash images, sounds . . . no sense to it at all. Just whatever worked, you know, in the sense of getting the viewer to watch a little more. They use eye tracking, you don’t even have to click. But every so often, whoever’s behind it—whoever generates these edit streams, assuming, that is, that there’s a human anywhere in that loop—will put it into the minds of those people, and then bad things can happen.”
Of course, the idea of algorithmically generated images finding just the right combination of colors to completely reprogram people is a bit silly. I hesitate to completely dismiss the general idea, though, given Elsagate
Elsagate was a phenomenon involving of a lot of very weird Youtube videos. A group of creators had been experimenting with various video formats, and one day a couple of them threw on 5$ Halloween costumes of Elsa, Spiderman, and other popular characters, and then acted out a bunch of slapstick scenarios. The video blew up on Youtube Kids, and so they, operating as a views-generating group mind / algorithm, pumped out more of these videos.
Inevitably, other channels jumped on the moneywagon, thus creating the “Elsa Spiderman” genre of video. Relatively harmless at first, but eventually these views-generating algorithms needed to stand out from the crowd, as the space got flooded and overall views went down through sheer supply saturation. At that point the videos started getting weird. Frequent themes would includes pregnancy, pooping, and simulated sex, because videos featuring this got more views and better view times. Why? What about Elsa fake pooping out plastic balls enthralls children of all cultures? Is that a real-life example of a Nan-Shub? Towards the end of the saga, semi-automated animation studios had been plugged into the genre, and the speed at which shit got freaky went exponential.
I read a random reddit comment as it all went down that described the Elsagate videos, and the characters within them, as “Psychofauna we don’t understand, feeding on our children’s minds.”
I’ve been wondering if this will keep happening now that image and text generation AIs have become widespread. Apparently, boomers are going coo-coo for cocoslop. Something about the form of them seems to draw them in and hold their attention, moreso than non-generated images. Is it hard to believe that algorithmic generation loops would stumble upon combinations that reach deep into our brains and enthrall us?
That’s of course a far cry from actually influencing our behavior. Cory Doctorow is skeptical of this theorized magical power that machine learning has given the big surveillance companies like Facebook. He describes the mystical stories of ML ad-targeting as just another aspect of advertising - of course Facebook wants people to believe their feed algorithm is near magical, because then advertisers are willing to spend more to get on it! He writes:
a platform’s job is to make these connections, and that means platforms are always in the algorithm business… the platform has to make a call as to what you’re going to see or do.
These choices are enormously consequential. In the theory of Surveillance Capitalism, these choices take on an almost supernatural quality, where “Big Data” can be used to guess your response to all the different ways of pitching an idea or product to you, in order to select the optimal pitch that bypasses your critical faculties and actually controls your actions, robbing you of “the right to a future tense.”
I don’t think much of this hypothesis. Every claim to mind control – from Rasputin to MK Ultra to neurolinguistic programming to pick-up artists – has turned out to be bullshit. Besides, you don’t need to believe in mind control to explain the ways that algorithms shape our beliefs and actions. When a single company dominates the information landscape – say, when Google controls 90% of your searches – then Google’s sorting can deprive you of access to information without you knowing it.
I share his skepticism at least insomuch as I don’t trust platforms’ claims about their own algorithm, but I do believe that we’re being beset upon by psychological paperclip factories that we as individual humans don’t have techniques to resist. At minimum, the constant assault is reducing our quality of life. At worse, it’s driving our entire existence - convincing us to participate in excessive consumerism, driving us to hate our fellow countrymen or people we don’t know and never will meet in other countries, or leading us to suicide. It’s also causing us to turn our existence towards the purpose of platform profit generation: FOMO causes us to take more pictures that we upload to Instagram for AI training while at the same time increasing the value of the platform by adding to the amount of content on it, while at the same time being advertised to! Every post on Reddit helps the Reddit sales pitch to advertisers and LLM companies to whom they sell your post for further training. Every Tweet, eaten by Grok, making a trillionaire even more billions.
There’s pushback. There’s an “analog” movement of people opting into dumbphones, or abandoning streaming services for MP3 players, or switching to e-ink notetaking devices to try to get themselves away from the distraction of their computers. Of course, capitalism noticed, and has infested the scene with consumerism. The message of “try this dumbphone out to help you avoid doomscrolling” has turned into “you NEED to buy this to fix your social media addiction! You can’t do it on your own, but this one last purchase will be the thing that finally solves it.” Inevitable. Always the cycles: hook you on the dopamine loop, then sell you what you need to get unhooked; but of course, the medicine doesn’t actually work. I’m reminded of the fact that antidepressants are some of the most profitable medicines to sell, since they don’t actually cure anything, they simply are a life-long treatment for a disease that they might not even be effectively treating. (That’s not a call to stop taking your psychiatric meds; listen to your doctors, kids)
I want to believe we have all the tools we need today to save ourselves from social media FOMO, doomscrolling, and dopamine holes, in the form of “willpower,” but experience is teaching me the importance of structural effects in managing my ADHD. If leaving my gym shorts next to my bed makes all the difference between me going to the gym or not in the morning, maybe it’s true that we just need to have a phone that can’t launch Instagram at all. If nothing else, it seems that willpower is a limited resource, and the need to constantly spend it every time we look at our phones is draining our ability to apply it to other situations throughout the day. For me, the application of bog-standard addiction management techniques helped me break my addiction to reddit, so maybe this can be generalized to the population?
Or, perhaps activists can fight fire with fire. The documentary filmmaker Kalle Lasn, known for launching campaigns such as “Buy Nothing Day,” tried this strategy. He created “anti-ads,” or “mind-bombs,” such as an ad that simply describes that “over 50% of the calories from a Big Mac come from fat.” An obvious issue stems from the fact that we don’t own the platforms. When Lasn tried this, not only was it expensive, the major networks eventually simply refused to show his ads. In Canada this is illegal, but that’s no matter, the Supreme Court of Canada simply refused to hear his case, thus solving any anxiety from the networks.
Micah Bornfree characterizes the psychic assault as “mental pollution,” and hopes to use this framing to allow for the same techniques governments use to wrangle corporate environmental pollution, to also wrangle mental pollution (I believe the idea originates with Michel Serres). I like his characterization, especially of how corporations use pollution as a weapon:
The Earth is being claimed by corporations. Whether it is by appropriating the ocean with their oil spills or by seizing public spaces with their advertisements, corporations use hard and soft pollution to steal what is ours.
I’m skeptical, though, because it’s not like we’re doing a great job at preventing corporations from setting the world on fire, especially during the AI era. The needs of the Datacenter overrule the petty needs of the humans.
Solution?
One of my favorite things that happened this decade was when Musk started selling Twitter verification badges, and people immediately used these badges to impersonate several large companies, for example posting from an impersonated Lockheed Martin account that the company would no longer be selling weapons to Israel, thus wiping billions off their stock value within minutes. That’s a BEAUTIFUL form of direct action: low effort, cheap, able to be performed by basically any individual anywhere, and enormously impactful. With LLMs, I wonder if a similar opportunity exists for direct action.
In the aforementioned “Fall, or Dodge in Hell,” the memetic, chaotic, and useless form of the internet was caused a decade earlier in the book by a character that released a series of FOSS bots that anyone could deploy that could, given a topic or person, seed an insurmountable amount of disinformation about said topic or person, thus making it impossible to separate fact from fiction. I don’t like the idea of destroying the entire internet, but I do like the idea of nuking platform value. Perhaps there’s some way to leverage LLMs to fill the platforms with so much useless, boring - yet just plausible enough to get past the algorithm - noise, that people close Instagram in disgust as it fails to deliver the ever-promised dopamine hit?
Doctorow is probably right that Facebook isn’t casting magic spells on us, but there is something new happening here. The entire psychology-of-advertising industry exists around the premise that there are as of yet undiscovered techniques to get people to buy your product instead of your competitors’, or to look at your social media app instead of the other one, or to click your ads first. As these people throw Machine Learning at the problem, and overwhelm us with the results of their studies, what have we been doing to resist this? Is there a psychological equivalent to a suit of armor that we can don to reduce the effect of the assault on our minds? Is there a memetic equivalent of a healing potion, something we can create for ourselves to “reset us to baseline” at the end of long day of being psychic assault?
Practically speaking, we can redesign our environment, of course. We could use dumbphones, put our phones in black and white mode, uninstall Instagram, install ad blockers, put pi-holes on our network to block unwanted content. Somehow this feels more like hiding, though. I wonder if there’s something we can do to face the assault head-on, something to regain control of our willpower and free will.