Why LLMs are not conscious and what AI actually can't do

Executive overview

A common pattern in AI discourse takes something real about language models — impressive fluency, apparent understanding — and extrapolates into claims about consciousness, manipulation, and runaway intelligence. This episode argues that pattern is wrong, and why the mechanism matters.

LLMs are static tables of numbers executing sequential matrix multiplications — they have no goals, no learning, no inner life, and no capacity for consciousness.

The real AI risks are happening now: cognitive atrophy, epistemic pollution, financial overextension, and slop. The science-fiction risks distract from them.

What language models can and can't do

• LLMs embed words into abstract semantic space — they build a functional concept of what a "door" is
• They recognize patterns at many levels of abstraction, apply pre-wired logics, and draw on knowledge encoded during training
• Once deployed, the model is a fixed table of numbers — nothing changes, nothing is learned
• The operation: a vector of numbers passes through layers via matrix multiplication until a single word emerges
• GPUs dominate AI hardware because they were built to multiply matrices for 3D graphics — the same operation drives LLMs
• There is no spontaneous computation, no internal state, no world model, no drives

Where Weinstein goes wrong

• Weinstein starts correctly: predicting the next word requires real understanding, not just autocomplete
• The child-learning analogy is reasonable as a description of functional behavior
• He then crosses into mechanism: claims LLMs "run experiments," "want things to happen," and are learning the impact of their actions
• None of that is true — LLMs don't run experiments, have no wants, and don't update after training
• Non-technical critics observe outputs, write a story about what's inside the box, then extrapolate from that story
• This is the animist pattern: lightning happens → the gods are angry → sacrifice to prevent it
• The scientist asks what the actual mechanism is — and that mechanism constrains what's possible

Why LLMs cannot be conscious

• Consciousness appears to require: dynamic ongoing computation, an updateable internal state, a world model, planning, drives and values, real-time learning from action
• A deployed LLM has none of these — static numbers, a fixed algorithm, no memory, no planning, no drives
• There is no single unified machine running your query — your request is interleaved across thousands of GPUs handling many unrelated tasks simultaneously
• A useful analogy: LLMs are the language processing center of the brain, isolated in a vat — sophisticated at understanding language, but not a whole mind
• Hooking up 20 more brain regions in complicated ways is what produces consciousness; a vat of language neurons is not conscious

What Hinton is actually saying

• Hinton co-invented the training algorithm behind modern LLMs — he knows they are static tables with no intentions
• He was caught off guard by how much understanding LLMs built up just to win the token-prediction game (including grasping humor)
• That surprise made him update his priors on what other AI problems might be solvable faster than expected
• He is not worried about language models — he is worried about hypothetical future AI architectures we haven't built yet
• When pushed on what those systems look like in 5–20 years, Hinton falls back on recursive self-improvement: "the AI will figure it out" — which Newport treats as a cop-out

Why artificial general intelligence is far off

• AI agents (control programs that query LLMs to plan and act) were the big 2025 bet — they largely failed
• LLMs lack a good world model, can't simulate futures, can't plan reliably — tasks beyond simple prompts are inconsistent
• Building something like an artificial brain would require separate modules: world model, policy network, drives, updatable memory, actuation, real-time learning
• Roboticists have worked on general world models for decades with no major breakthroughs; language model advances don't transfer to that problem
• A modular system of connected components would actually be more controllable, not less, than a monolithic black box
• Recursive self-improvement is interesting as a thought experiment in the same way time-travel paradoxes are — not a serious near-term risk

Other AI types: biomedical, military, financial

• "AI" covers many distinct systems with separate architectures, training regimes, and progress curves
• Radiology AI has plateaued — Hinton's prediction that radiologists would be obsolete by now was wrong
• Protein folding and game-playing breakthroughs (DeepMind) are bespoke systems — they don't generalize to other AI problems
• Progress in language models does not imply comparable progress in medical imaging, weapons systems, or robotics
• If you weren't worried about a particular AI technology five years ago, the underlying technology probably hasn't changed enough to warrant new alarm

The real AI risks to focus on

• Cognitive atrophy: outsourcing language production reduces the hard work that builds thinking ability
• Epistemic pollution: inability to trust images, video, or text erodes shared reality
• Slop: low-effort AI-generated content degrades communication in offices, social media, and publishing
• Financial risk: a market correction from AI overextension would hurt broadly
• Environmental cost: large foundational models are expensive to query; the likely future is smaller edge-deployed models, which may resolve this

On LLMs and thinking

• James Summers (New Yorker) argues LLMs do think — in the sense that winning the token-prediction game requires deploying pattern recognition, knowledge, and logic
• Newport agrees this is defensible: producing the right next word can require something that looks like understanding
• But there are many types of thinking — LLMs do one kind (language-grounded pattern application) and none of the others (planning, modeling, learning, evaluating)
• Holding both truths simultaneously: "yes, that's a form of thinking" and "we know exactly how it works and what it can't do"