Why LLMs won't power the brains of our machines
AI needs another breakthrough to conquer the real world
The transformer architecture powering generative AI models like ChatGPT is perhaps one of the most surprising discoveries of this century1. In simple terms, these models process sequences of text by breaking them into tokens, which represent words or characters. The transformer architecture is a stack of attention layers that analyse the relationships between these tokens—considering how often they tend to appear within the same context and their relative order. Based on these learned patterns, the model predicts the next likely token, appends it to the growing text, and repeats the process until it reaches a predefined length or generates a specified end token.
What has astonished both researchers and users is that this relatively straightforward approach produces human-like responses across a wide range of tasks. LLMs can write code, draft coherent essays, and even solve math problems. More surprisingly, the ability of these models to store knowledge, reason, and generate accurate outputs seems to improve as more data and computational power are applied during training.
This phenomenon aligns with empirical scaling laws suggesting that performance in language models continues to improve predictably as the amount of data, model size, and training compute are increased2. But while these scaling trends have driven LLM advancements, they also hint at limitations. As companies scramble for more data, GPUs, and energy to fuel these models, questions arise about how sustainable this exponential growth is. One study suggests that by 2028, we may exhaust all publicly available human-generated data3 . We are also seeing a cooldown of the frenzy around surpassing human-level intelligence. While the models have gotten better, anyone that has worked with them closely recognises the awkwardness of having to do "prompt engineering" to get them to perform tasks reliably.
In stark contrast, the human brain outperforms even the most advanced LLMs in general reasoning ability while consuming just 25 watts of power and learning from a tiny fraction of the data. While LLMs demonstrate remarkable general intelligence they still fall short of what we intuitively think of as human intelligence. And it’s unlikely that simply scaling up current models will bridge that gap.
LLMs as Q functions
In reinforcement learning (RL), two main paradigms exist: model-based and model-free RL. Model-based approaches train a model of the world, allowing agents to simulate actions and their consequences before committing to a decision. This ability to plan and reason about future outcomes gives model-based systems a distinct advantage in tasks requiring foresight, logic, or complex decision-making. Although these approaches can struggle in high-dimensional or complex environments, they often converge faster and with fewer samples than model-free methods.
In contrast, model-free methods, like Q-learning, bypass this simulation step by directly learning a function that maps states to optimal actions based on past experience. These systems excel in tasks where vast amounts of data are available, but they require significant training to perform well. They also tend to lack the flexibility to plan ahead or adapt dynamically to novel situations.
LLMs operate much like a model-free Q function, where the input is the sequence of previous tokens, and the action is the next predicted token. The action space is vast (typically ~100k-1M tokens vocabulary), and the objective is to produce sequences that align with the learned patterns in the training data and fulfil the user's instructions.
However, just like in model-free RL, LLMs don’t possess an internal model of the world or knowledge they encode. In fact, transformers don't store any state at all. They simply compute the next most probable token based on statistical relationships without simulating or understanding the implications of what they generate. This limitation makes plain LLMs poor at tasks that require deep reasoning or multi-step problem-solving, as they are prone to hallucinations. These limitations are particularly evident in tasks that require state-tracking or long-term reasoning, such as solving complex math problems, evaluating code, or following entities in a narrative4.
Some clever workarounds attempt to mitigate these weaknesses. For example, prompting LLMs to "reason out loud" by generating a step-by-step explanation before arriving at a conclusion mimics the planning ability found in model-based systems. This technique effectively builds a temporary, explicit model of the answer and has improved LLM performance in tasks requiring logic and reasoning. OpenAI seems to have doubled-down on this approach with their new O1-series models. Other Grounding methods can also help reduce hallucinations, improve factual correctness, and provide updated information. These involve techniques like injecting external information into the prompt and using the LLMs to judge their own outputs.
While these solutions improve the quality of LLM outputs and are driving productivity gains across many industries, they remain stopgap measures. The core limitations of LLMs as model-free systems persist. Without an internal model to plan or simulate future outcomes, LLMs will always struggle with reasoning tasks. This highlights the potential value of model-based approaches in future AI development, as these systems could provide the kind of structured reasoning and planning that LLMs currently lack.
What can we learn about biological brains?
Daniel Kahneman in the book Thinking, Fast and Slow exposes that human thinking operates through two distinct systems:
System 1 (Fast Thinking): This system is intuitive, automatic, and operates quickly with little or no effort. It relies on heuristics (mental shortcuts) to make decisions and judgments.
System 2 (Slow Thinking): This system is more deliberate, analytical, and requires effort. It is used for complex problem-solving, reasoning, and making decisions that require attention and logic.
LLMs are a purely System 1 brain, even though they can be made to perform System 2 tasks to a certain extent with tricks like reasoning out loud. A System 2 brain requires a different cognitive infrastructure, one that has feedback loops and can maintain an internal state throughout multiple inputs. Such architectures are able to support a model of the external world and keep track of its dynamics as we act on it, similar to the model-based paradigm in reinforcement learning. The transformer architecture lacks any of that: it's a purely feed-forward, single-pass, stateless function.
Human brains exhibit remarkable efficiency when learning from minimal data. We can often generalize from a few examples, adapt rapidly to new environments, and self-regulate based on past experiences. This stems from the brain's ability to maintain a rich and dynamic internal model of the world, constantly updating based on sensory inputs and prior knowledge. Internal models allow us to make predictions about future states, simulate hypothetical scenarios, and learn from unexpected outcomes, all of which are essential functions for adaptive behaviour. LLMs, in contrast, lack this persistent state and the ability to dynamically adapt over time in a self-supervised manner, requiring instead internet-scale datasets to be able to generalise.
Another critical aspect is meta-cognition —the ability to reflect on our own thought processes. Human brains constantly engage in self-monitoring, detecting errors, adjusting strategies, and learning from feedback in real-time. This allows us to course-correct even when we're unaware of the specific error we made. Current AI models lack this higher-order cognitive function. They do not possess intrinsic awareness or self-monitoring capabilities, which means that although they can simulate reasoning, they cannot reflect on or improve their reasoning independently. In other words, they are not aware of the meaning or effect of their output, nor whether it is correct or an hallucination.
In essence, biological brains are fundamentally wired for real-time adaptability, self-regulation, and efficient learning from limited data, while LLMs remain dependent on vast datasets and external supervision for improvement. The challenge for future AI systems lies in mimicking these organic feedback mechanisms and integrating a more persistent, dynamic model of the world.
Towards building truly autonomous machines
Perhaps the need for an architecture capable of System 2 thinking is not so apparent for LLMs in the way they are used today —in turn-based interactions with static input and output domains (text, images, videos, audio). However, the next obvious step for AI is to conquer the realm of manual labor, where the challenges are entirely different. In this domain, the inputs are live sensor feeds, often noisy and unpredictable, and the outputs are physical actions that must be precise and adaptable in real-time.
Reasoning in text tokens, as LLMs currently do, would be a poor substitute for understanding the complex, continuous dynamics of the physical world. The speed at which physical tasks unfold would require an AI that can think on its feet —maintaining an internal state, anticipating its sensory inputs, and responding in real time. Without this, hallucinations common in text-based reasoning could easily translate into nonsensical or even dangerous actions in a physical context. For example, a robot given an illogical plan or failing to adapt to real-world feedback could misstep, damage property, or cause harm to humans.
Additionally, there's a vast gap in the data needed to train such systems. While LLMs thrive on vast amounts of text or media from the internet, the data needed to teach an AI to operate in the physical world are far less abundant. The internet is a repository of knowledge but offers little in the way of lived experience: first-person sensorimotor experiences and embodied learning from tasks like walking, grasping, or manipulating objects. Humans acquire such skills through years of embodied interaction with the world, developing an intuitive physics engine based on constant trial and error. Collecting and training LLMs on this kind of real-world data would be prohibitively costly and time-consuming, requiring a fundamental shift in architecture and training paradigms.
Wrapping up
Training an AI model to live among humans with the transformer architecture would likely be both inefficient and impractical. The physical world demands not just reasoning but real-time adaptability, an intuition of physics, and the ability to learn from embodied experience—traits that are far from being achieved with today’s LLM-based AI. A shift toward architectures that integrate dynamic internal models, predict and anticipate the effects of their actions, and self-correct when the unexpected happens, are necessary for AI to truly move from passive digital agents to active participants in the physical world.
Transformers were introduced in the paper Attention Is All You Need.