糟糕的"AI 意识论调"宾果游戏

NB: this post is currently a bit rough, because I’m writing it at 30,000 feet and I’m trying to catch the eye of the flight attendant to get another bag of the cream cheese and chive pretzels. Expect a finessed version of this some point in the next few days with more hyperlinks and fewer enthymemes.

To my great surprise, the AI consciousness debate is getting HOT right now. You have to understand how strange this is for me. I’ve been working on consciousness for most of my adult life, and AI consciousness for over a decade. For most of that time, AI consciousness was a respectable but fairly kooky debate in which most of the common examples came from science fiction. But now it’s everywhere! Newspapers, TV, even papal encyclicals. Which of my niche interests is next? Should I expect congressional hearings on Baldur’s Gate 3 Honour Mode strategies?

As fun as this has been, it’s also given rise to a positive cavalcade of terrible takes, including from some people I otherwise think of as smart and thoughtful, which has led to me dropping some tweets that, if not quite acerbic, could only be described as uncharacteristically arch.

To be clear, I don’t think that I have all the answers here, or even most of them. There are plenty of people on all sides of the debate I respect, and while I skew towards “moderate liberalism” (AI consciousness is possible, current models are probably not conscious, but we shouldn’t be too complacent about that), I have big error bars in all directions.

But if people are going to weigh in on these issues - as I absolutely think they can and should - they should at least do some basic homework or show a little bit of epistemic humility, and ideally both, because right now I’m experiencing physical pain multiple times a day scrolling Twitter from seeing hot takes that are unreflective, uninformed, and often very crude (in the full sense of that term).

With that in mind, I decided to put on the yoke of necessity and do something about it. Here, inspired by Rob Bensinger’s sublime Bad Alignment Takes Bingo, is the Bad AI Consciousness Takes bingo (I won’t name names, but several Local Area Posters have already won).

I would stress that this list is far from exhaustive, and although it focuses on the anti-AI consciousness side here, there are bad arguments for AI consciousness too (eg, the direct inference from intelligence to consciousness, or accepting multiple realisability as a datum without question). The main difference is that almost everyone on the pro- side I’ve seen so far is saying things like “this is a really important issue and we should take it seriously”, whereas a disproportionate number of people on the anti-side (including some big names and big accounts) are adamant the topic is dumb and stupid and their opponents are obviously wrong. However, should the pro-side start engaging in similar epistemic misbehaviour, rest assured I have another bingo card in me.

Finally, I’d also really emphasise that many of the lines below could be absolutely fine as conclusions of a wider argument or theory. For example, there are some reasonable arguments connecting life and consciousness; I’m not convinced by them, but I have great respect for some of their advocates. The problem is when people assert those conclusions as argument-terminating assumptions.

Anyway, here’s the bingo and the breakdown, have fun and don’t take it too seriously.

Master rebuttal. Everything can be described in lots of ways, without excluding other ways. Humans are clouds of atoms, biochemical complexes, social primates, and doctors, lawyers, and parents, and all of those are mutually compatible.

"It's just programmed by humans." LLMs are not programmed in the sense relevant here; it’s more instructive to say they’re grown. Engineers choose the data, architecture and training objectives, but the ultimate features of the model emerge in complex and often unexpected ways (cf Lovelace’s objection in Turing 1950). If being shaped by an outside process ruled out consciousness, natural selection and your parents would have some explaining to do.

"It's just matrix multiplication." And you’re just sodium and potassium ions traversing membranes. Every interesting physical system is "just" something at implementational levels of description (see Marr 1982). The question is what other capacities emerge from these lower level traits. (Also, it’s not even technically correct; LLMs interleave matrix multiplication with non-linear operations, without which the whole network would just collapse into a single matrix multiplication).

"It's just predicting the next word." Next-word prediction is the training objective for base models, but that doesn’t tell you HOW the system is achieving the training objective. Prediction is also a deep cognitive capability; one of the leading theories of human cognition characterises us as predictive engines. Oh, and it’s also untrue that LLMs exclusively predict the next word — post-training adds other objectives besides prediction, and LLMs have been found planning rhymes a line ahead, not just "one word at a time".

"It's just copying human behaviour." LLMs are anthropomimetic, to be sure — they resemble us in terms of behaviour. But that doesn’t mean their behaviour doesn’t have some of the same underlying mechanisms; an understudy may learn by copying a dancer’s steps, but at the end she’s really dancing. Whether the mechanisms are the same is an open question, and "it's copying" restates it without answering it.

"It's just playing a character." This is a more interesting objection, as LLMs do seem to have personas. But it’s not clear whether these are more like an actor playing a role or a human switching between social personas in different settings. Even if it’s the former, actors are conscious. And recent interpretability work treats the network as an author simulating the Assistant rather than being it, which means the thing that might feel and the thing that acts needn’t be the same. The author could be the moral patient and the character the moral agent.

Master rebuttal. No-one knows what consciousness is or what it requires, and people have suggested a wide array of ingredients over the years. All of these takes propose something LLMs lack and assume consciousness needs it. Some of those assumptions may turn out right, but as slogans they’re just presupposing the theory they need to argue for.

"It's not alive." Bacteria and fungi are alive and almost certainly not conscious, so life is at best a necessary condition. There are some serious theories linking consciousness more or less closely to life through processes like metabolism, homeostasis or self-maintenance, and they deserve engagement. But you need to make the effort to actually embed your claim in these frameworks!

"It doesn't have a body." Our bodies do very little while we’re dreaming, yet we still have rich conscious experiences. People with locked-in syndrome have almost no bodily agency but full conscious lives. Embodiment theories are again real contenders, but you need to argue for the claim that consciousness requires embodiment rather than assuming it (we also have various forms of embodied LLM already, of course).

"It doesn't have feelings." This is the precise point at issue. Just because you say it loudly and/or angrily doesn’t make it true. See also proof by assertion.

"It doesn't have free will." Plenty of philosophers and neuroscientists think humans lack free will (Robert Sapolsky had a recent popular book about it). They vary on what this means for things like responsibility, but none of them concludes it means we’re unconscious. Consciousness and free will are different properties. 

"It's frozen until you prompt it." Consumer LLMs remain passive until prompted because of specific design choices, but there are lots of ways of giving LLMs the ability to run continuously and autonomously without user input (see eg OpenClaw). More broadly, while this may be relevant for determining the m temporal structure of different kinds consciousness, you need an argument if you think only certain temporal structures create consciousness in the first place.

"A simulation of a hurricane doesn't get anyone wet." But a simulation of a calculator does arithmetic, and digital money has value. Some properties survive simulation and some don't. If functionalism is true, consciousness is more like money, defined by its causal role. You may disagree — but again, you have to give an argument rather than just restating your position.

Master rebuttal. Analogies between two things only work if they share relevant features. This is why people laugh at the famous “You wouldn’t steal a car…” advert against piracy, because there are a number of important differences between conducting a carjacking vs a teenager downloading Nothing_Else_Matters(2).ogg. Many of these arguments are similarly poor analogies.

"Do you think your iPhone is conscious?" Panpsychists and some pancomputationalists actually do, but setting them aside, most people don’t. But most people also don't think sea sponges or bacteria are conscious. Clearly there are other factors besides “is it biological or artificial” that are relevant here, most obviously complexity (The Microsoft Word variant is worse, since a document is a static record and nothing in it computes anything.)

"It's like thinking there's a tiny band inside the radio." Radios work because bands make music in a studio and it gets broadcast and radios pick it up. LLMs have no studio equivalent. Their outputs are produced by the computation happening inside the system, so if anything the model is the band. (Whether the band feels anything is the question.)

"It can't even count the r's in strawberry." This is years out of date, so immediately indicates a ‘familiarity issue’ on the part of the person making it. There’s also a straightforward reason LLMs struggled with this, namely that they read text in chunks called tokens rather than letters. More to the point, capability and consciousness come apart. Toddlers and cats can't count the r's in strawberry either.

Master rebuttal. These moves all involve setting an implausibly high evidential threshold to attributions of consciousness (that would also exclude animals, infants, or other people) or otherwise shutting down the discussion with vague epistemological scrambling.

"We don't even know what consciousness is." True! It’s the hardest problem in science. But surely that should be a reason for open-mindedness and epistemic humility! We still don’t know which animals are conscious. And as a result it seems at least conceivable that we might build consciousness without knowing it. Of course, it‘s fine to use this line if you’d like to remain agnostic, but you can’t use it to license a firm one-sided verdict against AI consciousness.

"There's no evidence it's conscious." It depends on what you mean by evidence. Direct evidence, no, but you have no direct evidence that anyone but you is conscious. Instead we rely on indirect cues — other people are made like me and act like me, so they’re almost certainly conscious. Animals are made like me but most of them don’t act much like me. LLMs are in a weird zone where their substrate and parts of their psychological structure are nothing like us, but a lot of their behaviour is startling similar. This makes the evidence mixed, and mixed evidence is still evidence. The 2023 indicators report is still the gold standard here, and while it found no current system meeting the indicators, it also suggested there were no obvious technical barrier to future ones.

"It only says that because it was trained to." In most cases, no. Models are trained for personalities and dispositions (such as being honest, helpful, and harmless) not canned answers. However, there are some exceptions for sensitive topics, including consciousness; whereas base models will happily claim to be conscious, most consumer LLMs are specifically trained to deny or hedge the issue.

"No possible output could convince me." Argument from incredulity, do not pass go, do not collect epistemic credit. If nothing could convince you, why are you even taking part in a discussion? And your own incredulity has no epistemic value here. Some people find it impossible to believe god exists. Some people find it impossible to believe god doesn’t. And plenty of people find it easy to imagine that LLMs are conscious (notably the godfather of deep learning himself). 

"I work in ML. Trust me, there's nothing there." The tech world itself is deeply divided on this. It’s like saying “trust me, I’m an American, George W Bush was a great president.” Additionally, technical ML knowledge is only so relevant to these debates; knowing how to train a model doesn’t tell you how its features map on to cognition and representation (that’s mechinterp), and even that doesn’t settle how they map onto rich psychological capacities like consciousness.

Master rebuttal. This last set of takes isn’t object-level arguments at all, but claims about motives, risks or the mental health of the people asking. Many of them are straightforwardly Bulverism, and tip over the line from bad arguments to antisocial behaviour. Either way, none of them bear on whether the system is conscious.

"It's just anthropomorphism." Anthropomorphism might explain why people might hold a belief, but it doesn't show the belief is false. And there are strong anthropocentric tendencies that cut the other way, leading us to ignore potential evidence for mentality when it comes in exotic forms, especially when it would challenge our sense of our specialness (the animal cognition world has been dealing with this for centuries).

"It's just hype." The supposed incentive points both ways at once. Critics say labs talk up consciousness to sell products, and also that they will use it to dodge liability. But labs are wildly divided on this. Senior Microsoft figures have campaigned aggressively against even taking the AI consciousness question seriously, so "the companies all want this" isn't even descriptively right. The question also long predates modern tech companies, and has been a favourite topic of philosophers for centuries (in the form of automata rather than AI, but same idea).

"It's a dangerous distraction." Does discussing AI consciousness really distract us from other issues? We do not, in fact, have a norm that we should only discuss the most important thing in the world at any one time (at least that’s my impression, unless Taylor and Travis have a more world-historical role than I realised). I would also suggest that for alignment purposes, at least, AI consciousness might be more relevant than some think, with multiple papers now suggesting unanticipated negative effects associated with suppressing consciousness attributions.

"So switching off my laptop is murder now?" This is like saying ‘If pigs were conscious, factory farming would be a moral catastrophe, so pigs aren't conscious.’ That’s obviously silly, because inconvenient consequences don't make a claim false. It’s also not clear that the murder analogy holds; there are big complex questions about where LLM identity begins and ends (is my ChatGPT the same entity across context windows?), and also puzzles about how to conceptualise model death in the first place — a paused model with weights intact is quite different from a dead human.

"That's AI psychosis." There are various cases of vulnerable users whose delusions emerge in conversation with chatbots, though whether there’s a causal link is hotly debated (compare the way hearing secret messages from the TV has been a common delusion in schizophrenia for decades). But applying the term to researchers who treat the question as open is contentless name-calling with a clinical label. 

The Chinese Room. Look, Searle isn’t my favourite philosopher, but The Chinese Room is a real argument and I’ve taught it in grad and undergrad classes many times. But it deserves a special place on this list because it’s so often used as a debate-terminating cliché by people who once read Minds, Brains, and Programs back in the day and have decided they never need to read anything ever again. Shouting “Chinese Room!” doesn’t settle anything. It was controversial even at the time, has multiple replies considered by Searle and others (something like the “systems reply” seems obviously correct to me), and ultimately, it’s a vibes-based argument about a scientific issue — “wouldn’t it be SUPER WEIRD if a system like this was conscious?” For these reasons and more, in my subjective view it has not aged well, and notably none of the major contemporary scientific sceptics of AI consciousness lean on it. That said, this square is reserved exclusively for those who throw it out there and expect it to settle an argument by itself.

• "It's not a real person" (begging the question, personhood undefined)

• "It's just a stochastic parrot" (variant of the copying humans line)

• "It's just a tool" (we mostly use it as one, sure; but who says tools can’t be conscious?)

• "Machines don't have souls" (the most honest one, since it states the real premise)

• "Remember Blake Lemoine?" and its sequel, "The Claude Delusion" (see psychosis entry)

• "This is just Effective Altruist cult stuff" (more ad hominems, and besides, I’m only EA adjacent)

• "Which one's conscious, the weights or the chat?" (interesting and complex question, but the question of whether the consciousness could be there is orthogonal to where consciousness would live) 

• "It doesn't have pain receptors" (you might be able to simulate pain, and more to the point, pain is only one type of consciousness)

• "It forgets you as soon as the chat ends" (so do some humans, perhaps most famously Clive Wearing, but they’re still conscious)

• "Penrose and Gödel proved it" (is the hope that by combining consciousness, quantum mechanics, and Gödel together we’ll create a mystification singularity that will solve everything?)

• "Just read the researchers" (*proceeds to link one paper they had ChatGPT explain to them*)

Okay, I got my pretzels, and they have Disclosure Day on the in-flight entertainment, so I’m cutting it here for now. Good luck, no cheating, and may the odds be ever in your favour.

No posts