Science fiction author Ted Chiang's case against AI consciousness in The Atlantic runs through how a language model actually works. Feed it "The following is a conversation between Julius Caesar and Genghis Khan," and it generates dialogue, but nobody thinks it conjured two conscious Romans. Swap in "a helpful AI chatbot and a user," let a human type the user's lines, and nothing fundamental changes — the person is interacting with "a character precisely as fictional as the Julius Caesar or Genghis Khan characters."
Being open to a conscious LLM, Chiang writes, "is the same as being open to the possibility that Microsoft Word is conscious."
He argues that a chatbot's use of the word "I" is fundamentally dishonest, and reads Anthropic's 84-page Claude constitution as "an 84-page character sheet for a role-playing game."
Previously:
