LLMs are Doing What We Do. Maybe That's A Problem. Maybe not.

Why Apple's big new deflationary result may not mean what everyone thinks it does

There’s been a lot of buzz this week about Apple’s new research paper, The Illusion of Thinking. The paper looks at how large language models (LLMs), even when given plenty of time to “think step by step,” tend to break down on complex reasoning problems. As difficulty increases in certain kinds of reasoning problems—think Tower of Hanoi, river-crossing puzzles—performance doesn’t just degrade. It falls off a cliff. More interestingly, the models actually use fewer tokens for harder problems, as if they’ve learned to give up rather than even attempt to reason.

Predictably, this has triggered a wave of anti-LLM sentiment. The usual critics (we hear you Gary!) are pointing to this as definitive proof that LLMs are fundamentally flawed—that without “real” reasoning, internal models, or symbolic manipulation, they collapse under pressure.

But maybe the critics have it backwards. This isn’t a flaw in the models—it’s a flaw in us, mirrored in the models.

LLMs are trained on human-generated text. They learn what we say, how we say it, and when we stop saying it. When human beings reason—especially in language—we don’t typically carry out long, flawless chains of logic. We cut corners. We jump steps. We stop halfway through. We compress, simplify, and bail out. Why?

As I have argued elsewhere, human thought itself is likely fundamentally autoregressive, just like LLMs. However, the reach of such autoregression in our case is inherently limited by biology. Unlike LLMS, which can retain an extremely long context perfectly while generating, human memory limitations mean that each successive “token” of thought is generated from a context whose influence fades with distance. As a result, we forget, get confused, change our minds. These constraints aren’t just inside our heads—they’re present in language itself, which is a product of those heads. The training corpus literally bakes in this distance decay. Attention-weight patterns in LLMs tend to diminish with token span because that is how human cognition operates. The model isn’t handicapped; it has simply learned the geometry of cognition as revealed by language.

So when a model stops, shortcuts, or simplifies exactly where humans would, it may not be an error. It may be an authentic reflection of how thought appears under our biological limits.

In other words, these systems didn’t just learn to think from human language. They learned to think like human language. They have learned to model the human cognitive system as revealed through the corpus of language on which they have been trained. And that corpus itself inherently encodes the limitations of the human cognitive system in processing longer-range dependencies.

In other words, the so-called “limitations” of LLMs may simply reflect them doing too good a job of learning the properties of the human-generated data.

Which leads to a potential revelation.

Humans don’t reason across long sequences because we can’t. Our brains, built on wetware designed to solve many problems simultaneously (you know, breathing, sensing, moving). LLMs are inherently under no such constraint. There is nothing stopping us from building models that learn from the language we made under cognitive pressure—but then operate with expanded memory and persistent context.

In other words, perhaps these models could become better than us at using the tools we ourselves created.

They can inherit our cognitive software, while freeing themselves from the hardware.

But this may turn out to be harder than it sounds.

Because here’s the deeper problem: we don’t have access to a version of language that isn’t shaped by human cognitive constraints. Every sentence in the training data—every explanation, story, argument, or line of reasoning—was generated by a brain operating under biological pressure. The limitations aren’t just in the model. They’re in the corpus. The data itself is the bottleneck.

So if we want to build systems that go beyond human cognition, we may not just need bigger models or longer context windows. We may need a different kind of training data altogether—or a different way of thinking about what language is and how it encodes thought.

That’s the real challenge. And maybe a real opportunity.

But let’s not forget: for most of the language humans actually use on a regular basis—in work, communication, even expert writing—the level of reasoning these models provide is more than enough. Most of the time, we’re not solving Tower of Hanoi in ten moves while sipping our Starbucks. For the kinds of tasks 99.9% of human language is currently used for—talking, writing emails, making decisions, drafting reports, or synthesizing other people’s writing—these models already deliver human (superhuman?) levels of performance.

And that might be the only level needed to dramatically change the face of the planet.

Edit: Postscript. Thanks to Maykyta (see comments) for raising a very important point that I should have, which inspired me to amend the title of this piece with the ‘maybe not’. The tasks used in the paper are all the kinds that humans themselves solve using extra-linguistic systems like visual imagination and even pen and paper. So yeah—this paper is kind of a BS way to claim you’re undercutting purely linguistic models. Yes, we need more than language to solve all cognitive tasks. But that’s a limitation of language itself, not language models. And the visual models—built on autoregressive engines(!)—-are getting incredibly good (see Veo 3). All we need is a bridge.