What LLMs can't do.***
Before getting to the limitations of LLMs, one must first address the relentless, half century long campaign to present LLMs (or “AI”) as “thinking”, “reasoning”, or “intelligen”. I wrote about this in an earlier essay The Mother of All Category Errors.
Large Language Models are not intelligent. To ascribe all the properties of intelligence to nothing more than a pattern matching algorithm is the biggest and most consequential “category error” ever foisted on Enlightenment civilization.
Here is a definition of category error:
A category mistake (or category error) is a semantic or ontological error in which things belonging to a particular category are presented as if they belong to a different category, or, alternatively, a property is ascribed to a thing that could not possibly have that property.
- Wikipedia, Category Mistake
I have placed a few critiques along these lines into the [APPENDIX].
----
With the canard of intelligence identified, let’s move on to the theoretical and practical limitations of LLMs as a technology.
[NOTE] This is the second essay in a series. The first essay is Lobotomy as a Service.
Seven Failure Modes
The world is presented to LLMs by prompts, which consist of data that is in the LLM training set, and data that is not. Each kind of prompt data has its own failure modes. Running the trained LLM produces both errors (hallucinations) and semantic ablation (ignoring the tails of the data set). Using data produced by an LLM to train a new LLM quickly leads to a failure mode called “model collapse”.
OTOH, when a prompt contains information that is not in the data set, the error rate skyrockets. A lot of what is not in the data set is called “tacit knowledge”.
In this essay, I will discuss seven limitations of LLMs.
Algorithmic Bias has been recognized and adjudicated for years, but the LLM crowd don’t want to add this to the charge sheet of LLMs.
Everyone has heard of errors (hallucinations), so I won’t beat that dead horse.
Nastruzzi’s discussion of Semantic Ablation explains why LLM output seems so bland and conventional. It is a very important “elevator pitch”.
Model collapse is important to be aware of because it is an effect that will become increasingly important as the internet is polluted with semantically ablated LLM outputs.
Memorization is a huge legal issue that gives the lie to the hucksters claims that LLMs are “thinking”.
Automation blindness is another well understood phenomenon that punctures the “human in the loop” talking point.
Finally, the failure to capture tacit knowledge (TK) is a replay of the failure of the “knowledge engineers” of the 1980s expert systems to translate vague, imprecise skills and intuitions into concrete rules. And of course, the proposed solution to this limitation is to revisit knowledge engineering.
1. Algorithmic Bias
That’s a fancy term for Garbage In, Garbage Out, which is computer science 101. If the training set is wrong or full of lies, the LLM can’t fix it. That’s because it has no world view. It does not think. Its training data is all it has. That a behavior so fundamental has to be highlighted shows a lack of historical awareness of the pitfalls of this technology.
1.1 Predictive policing
A classic example of this problem is the use of “predictive policing” as a corrective for racially biased policing. The theory goes like this: “Our cops are prone to “unconscious bias” when they choose where to patrol and whom to pull over or stop and frisk, We will feed the objective data about arrests into a predictive model and it will tell the cops where to go looking for crime.
BUT...the crime statistics that are used to train the model come from biased policing....The computer will tell you to go and find more Black and brown people to search, aand some of those people will have contraband, which wll generate more arrests, which will generate more data used to “refine” the model, which will tell you to do even more biased policing next time around.
- Corey Doctorow, The Reverse Centaur’s Guide to Life After AI
1.2 Oil price manipulation
The huge delta between the price of oil at the dock and the price in the financial markets is direct evidence that the market, or paper, price of oil is being manipulated. And, mirabile dictu, it is the adoption of AI algorithms that allow the manipulation.
(Hedge funds are) getting the trading strategy of (seasoned traders with 20 years of experience) and uploading it into a computer. Now what you’ve got is a bunch of algorithms, trading algorithms. They’re called algos and markers. And the algos react to text to news events. So, if you can flood the zone with headlines and news stories that say the straight (of Hormuz) is open, oil prices are coming down...
All of these terms will show up in those searches And what you do is you drive the algorithms absolutely insane....
But if you ‘ve ever traded markets, you know that if a whale comes in, you get crushed... So you can squeeze long positions out of the market by using these algorithms and getting enormous amounts of capital outlay into the short positions
Maybe AI doesn’t take over the world and turn into Skynet. Maybe it just creates like a failed centrally planned Soviet style economy that you can maipulate really easily.
- Trump paused war to manipulate oil prices (a video, with transcript)
Linking to the whole URL embeds the video, which I chose not to do. If you want to watch the video go to (https://www.youtube.com/watch folllowed by ?v=4QRpMjgE-pw)
2 Hallucinations
LLMs are competent regurgitators of the data in their training sets. Even then they make mistakes. To call these mistakes “hallucinations” is to use an anthropomorphizing euphemism for “errors”. Hallucinations are one thing about LLMs that actually are “inevitable”. They are baked into the algorithm - LLMs seeing stuff that isn’t there.
3 Semantic Ablation
Semantic ablation is the opposite failure mode - ignoring long tailed, high entropy, rich data. It is a “subtractive bias”. Like Hallucinations, its baked in to the LLM algorithm. Claudio Nastruzzi’s description of semantic ablation is so clear and concise that it has gone viral.
Semantic ablation is the algorithmic erosion of high-entropy information. Technically, it is not a “bug” but a structural byproduct of greedy decoding and RLHF (reinforcement learning from human feedback).
During “refinement,” the model gravitates toward the center of the Gaussian distribution, discarding “tail” data – the rare, precise, and complex tokens – to maximize statistical probability. Developers have exacerbated this through aggressive “safety” and “helpfulness” tuning, which deliberately penalizes unconventional linguistic friction. It is a silent, unauthorized amputation of intent, where the pursuit of low-perplexity output results in the total destruction of unique signal.
We can measure semantic ablation through entropy decay. By running a text through successive AI “refinement” loops, the vocabulary diversity (type-token ratio) collapses. The process performs a systematic lobotomy across three distinct stages:
Stage 1: Metaphoric cleansing. The AI identifies unconventional metaphors or visceral imagery as “noise” because they deviate from the training set’s mean. It replaces them with dead, safe clichés, stripping the text of its emotional and sensory “friction.”
Stage 2: Lexical flattening. Domain-specific jargon and high-precision technical terms are sacrificed for “accessibility.” The model performs a statistical substitution, replacing a 1-of-10,000 token with a 1-of-100 synonym, effectively diluting the semantic density and specific gravity of the argument.
Stage 3: Structural collapse. The logical flow – originally built on complex, non-linear reasoning – is forced into a predictable, low-perplexity template. Subtext and nuance are ablated to ensure the output satisfies a “standardized” readability score, leaving behind a syntactically perfect but intellectually void shell.
The result is a “JPEG of thought” – visually coherent but stripped of its original data density through semantic ablation.
If “hallucination” describes the AI seeing what isn’t there, semantic ablation describes the AI destroying what is. We are witnessing a civilizational “race to the middle,” where the complexity of human thought is sacrificed on the altar of algorithmic smoothness.
- Claudio Nastruzzis, Why AI writing is so generic, boring, and dangerous: Semantic ablation
4 Model Collapse
Model collapse is what happens when LLMs train on data produced by LLMs. After a few iterations, semantic ablation causes a complete collapse of the model. The most referenced paper on this topic is The Curse of Recursion: Training on Generated Data Makes Models Forget.
With the idea of LLM output as a lossy compression algorithm, its clear that recursive training is like taking a xerox of a xerox of a ... This intuition is literally what it looks like when LLMs that manipulate images are run recursively:
Ed Zitron has written extensively on this topic:
AI models, when fed content from other AI models (or their own), begin to forget (for lack of a better word) the meaning and information derived from the original content, which two of the paper’s authors describe as “absorbing the misunderstanding of the models that generated the data before them.”
In far simpler terms, these models infer rules from the content they’re fed, identifying meaning and conventions as a result based on the commonalities of how humans structure things like code or language. As generative AI does not “know” anything, when fed reams of content generated by other models, they begin learning rules based on content generated by a machine guessing at what it should be writing rather than communicating meaning, making it somewhere between useless and actively harmful as training data.
There is, of course, one blatantly obvious problem: generative AI models are prone to hallucinations when using human data. How does synthetic data, created by the very models that need to improve, improve the situation? What happens when the majority of a dataset is synthetic, and what if that synthetic data has within it some sort of unseen bias, or problem, or outright falsehood?
Are you seriously telling me that the solution to models giving the wrong answer to questions is to feed them more information from models with the same problem? Are you fucking kidding me? I don’t care that there are allegedly ways to make smaller models trained on limited synthetic datasets. That doesn’t fix the overall problem that we are running out of data to feed the models, and that the solution in many cases is to use the broken tool to make more data.
I am, of course, conflating two problems — the deliberate creation of synthetic data by AI companies and AI-generated content filling the internet with synthetic data that models are then trained on. Yet the end result is the same — forcefully teaching autocomplete typos in the hopes that it’ll be able to work out how to write America’s next great novel.
- Ed Zitron, Bubble Trouble
If your LLM is suffering from model collapse, an organization relying on that LLM output is also going to have serious problems.
AI is incapable by its nature of validating (fact, truth, accuracy), it’s worse than useless--it’s destructive on a system-wide scale.
The evidence of the systemic destruction is already overwhelming. Bogus “scientific papers” are already proliferating at an accelerating rate, making the task of identifying incorrect and fabricated (i.e. hallucinated by AI) data, processes and conclusions impossible due to the scale of the misinformation and the difficulty of identifying the misinformation buried inside superficially legitimate papers.
With both scientific and economic data and analysis now untrustworthy without exceedingly expensive, time-consuming vetting by human experts, where does this leave the “AI will automatically generate superabundance” hype? What’s already clear--but inconvenient--is the mass adoption of inherently flawed AI is undermining the foundations of “value,” however we wish to define it.
- C.H. Smith, AI’s Insurmountable Flaw: “Mass Regurgitation of Misinformation
5. Memorization
LLMs are a fancy and buggy search engine whose output is not trustworthy. LLMs do not “think”. Basically, they memorize and recite their training data. Which results in “casual plagirism”, covered by the rhetorical fig leaf that the LLM is “thinking” by paraphrasing. What is really going on is that the LLMs are a form of lossy compression.
AI does not absorb information like a human mind does. Instead, it stores information and accesses it...The resulting LLM is essentially a huge database of contexts and the tokens that are most likely to appear next...
- Alex Reisner, AI’s Memorization Crisis
A more impactful issue with memorization is that LLMs are being sued for copyright infringement. The AI companies have scraped the entire internet, copyright be damned. Owners of that ripped off intellectual property have demonstrated that they can get an LLM to regurgitate entire Harry Potter novels by prompting with the first paragraph. The name for this inconvenient behavior is “memorization”.
when prompted strategically by researchers, Claude delivered the near-complete text of Harry Potter and the Sorcerer’s Stone, The Great Gatsby, 1984, and Frankenstein,
Memorization could have legal consequences in at least two ways. For one, if memorization is unavoidable, then AI developers will have to somehow prevent users from accessing memorized content, as law scholars have written.
A second reason that AI companies could be liable for copyright infringement is that a model itself could be considered an illegal copy.
- Alex Reisner, AI’s Memorization Crisis
6 Automation Blindness
When management decides to fire staff and replace them with AI, they leave a tiny team as “humans in the loop” and claim that this will catch LLM errors. Psychological studies have proven this to be impossible because of “automation blindness”.
(If the hospital fires nearly all of its radiologists, the surviving human radiologists’ job goes from reviewing X-rays to Reviewing AI output. ...They become a “human in the loop”, charged with confirming judgments the AIs make at a superhuman clip.
This is an essentially impossible ask, thanks to well-understood concepts like “automation blindness”. When your job is to review something this is usually fine, you eventually lose the ability to spot when its not fine. The TSA proves this every day...”when government “red teams” test TSA security by trying to smuggle in bombs and guns, they almos always escape attention. The human sensory apparatus is just not built to maintain vigialnace for something that never happens.
- Corey Doctorow, The Reverse Centaur’s Guide to Life After AI
7. Tacit Knowledge
The first six failure modes have to do with prompts that falls within the training set.
The failure of LLMs to deal with situations that are not covered in their training sets exposes the gap between LLMs and “intelligence”. LLMs do not have a world view. Because they lack tacit knowledge they do not have common sense. Because they can not make any kind of commitment to the validity of their training set, they cannot detect lies or simple errors in it.
A large fraction of the holes in the training set are caused by a lack of tacit knowledge, which is what humans use in unique situations.
Tacit knowledge or implicit knowledge is knowledge that is difficult to extract or articulate—as opposed to conceptualized, formalized, codified, or explicit knowledge—and is therefore more difficult to convey to others through verbalization or writing. Examples of this include individual wisdom, experience, insight, motor skill, and intuition. An example of “explicit” information that can be recorded, conveyed, and understood by the recipient is the knowledge that London is in the United Kingdom. Speaking a language, riding a bicycle, kneading dough, playing an instrument, or designing and operating sophisticated machinery, on the other hand, all require a variety of knowledge that is difficult or impossible to transfer to other people and is not always known “explicitly”, even by skilled practitioners.
- Wikipedia, Tacit Knowledge
This problem is not new. The 1980s craze for expert systems tried to “extract” tacit knowledge by interviewing experts. It failed.
The decline of expert systems in the late 1980s was primarily due to their inherent limitations in scalability, adaptability, and interdisciplinary integration, which rendered them less practical compared to emerging machine learning techniques. Updating and maintaining the knowledge base of expert systems was a labor-intensive process, making it difficult to extend problem domains and limiting their usefulness.
Expert systems were also incapable of evolving beyond their initial knowledge base. Unlike modern machine learning methods that can dynamically improve from new data, these systems required extensive manual updates to adapt, resulting in rigidity that hindered their long-term viability.
Moreover, the interdisciplinary challenges in developing expert systems made it difficult to integrate them into diverse practical applications. Combining expertise across different fields often led to inefficient and less effective implementations.
- Robotics and AI blog, The Rise and Fall of Expert Systems in the 1980s
But that exact solution is what is now being touted as the solution to the gaps in LLMs.
I’m going to quote from a paper that defines TK, recognizes what a problem it is, and then, with LLM-like confident hallucination, goes right back to 1980s left brain thinking that we can somehow capture informal reasoning by some engineering scheme. The author’s scheme has at least thirteen steps that purport to pin down RK like a butterfly on a display felt. Been there, done that.
The author does understand TK and understands the issues that TK presents for LLMs. He correctly references Michael Polyani.
Tacit Knowledge (Non-codifiable)
• Pattern recognition heuristics developed through experience
• Embodied intuition from repeated practice
• Context-dependent judgment calls
• Sensory-motor coupling in physical domains
• Real-time adaptive reasoning under ambiguity
The latter category — what Michael Polanyi termed “knowledge we cannot tell” — remains structurally inaccessible to transformer architectures trained on static text corpora. This isn’t a scaling limitation that GPT-5 or GPT-6 will solve; it’s a fundamental constraint of learning from observational data that was never externalized.
…senior engineers consistently outperform GPT-4 on:
• Architectural decisions with long-term maintenance implications
• Debugging issues with subtle interaction effects
• Navigating organizational constraints and technical debt
• Making judgment calls on appropriate abstraction levels
Why? Because the reasoning behind these decisions was never written down. Code commits show what changed, not why specific tradeoffs were made. Post-mortems document failures but rarely capture the heuristics that prevented other failures from occurring.
the more expertise relies on tacit knowledge, the harder it is for AI to replicate without direct knowledge transfer from the expert.
He then goes on to revisit forcing a classic logical framework onto TK.
The bottleneck isn’t “learning to use AI tools” — that’s the commodity skill. The bottleneck is extracting tacit knowledge from expert minds and converting it into explicit, executable representations.
This requires meta-cognitive capabilities most practitioners lack:
1 Introspective Awareness: Recognizing your own unconscious decision-making patterns
2 Analytical Decomposition: Breaking intuitive judgments into discrete logical steps
3 Edge Case Enumeration: Articulating the boundary conditions of heuristics
4 Counterfactual Reasoning: Explaining why alternatives were rejected
5 Formalization: Translating fuzzy intuition into precise natural language instructions
He then predicts that the solution is the same old Knowledge Engineering.He calls them “reusable reasoning systems”. Its deja vu all over again.
Tier 1: Knowledge Architects
• Extract and codify their domain expertise
• Build reusable reasoning systems on top of AI
• Scale their judgment through automated execution
• Capture asymmetric value from knowledge multiplication
Tier 2: Tool Operators
• Use pre-built AI applications
- S. Bhattachargee, The Uncodifiable Advantage: Tacit Knowledge as the Strategic Bottleneck in AI Systems
The State of Play
We are at the exact same phase of the hype cycle that Expert Systems went through: yes, there are TK problems; but we will extract that knowledge by beating on the problem with logic and reasoning. LLMs are three years in on the same grift that Expert Systems ran in the 1980s. Telling people that the models will be perfected soon and the commercial applications will revolutionize the world.
No matter how expensive the failure, people keep wasting billions on non-functional “expert” systems. As late as 2015, IBM wasted $4 Billion on its Watson Oncology Diagnosis system.
Again, a lot to digest in this essay. But, the takeaway is that anyone peddling LLMs as a panacea is blowing smoke up your ass. Even when they recognize a problem, like TK, they go back to the same failed solutions. The Chinese approach of adding sensor and feedback data to the training is much more likely to produce some flavor of common sense that would work on a factory floor.
The next section will analyze why there is a massive investment in such brittle, flaky technology. Spolier alert: lazy, stupid, greedy management and tech evangelists with a hidden agenda.
***
The title is homage to Hubert Dreyus’s “What computers can’t do”. It also alludes to the never resolved, rancorous, philosophical disputations about the mind. My opinion is that in a debate, one should never allow LLM proponents to drag you into the philosophical quagmire of “what does it mean to be intelligent?”.
[APPENDIX] On Category Errors
AI does not fix a society; it reflects it. And whatever it reflects, it magnifies....No algorithm, no matter how sophisticated, can supply virtue where none exists...An algorithm can optimize efficiency, but efficiency is not wisdom. Optimization is not judgment. And judgment—moral, historical, human judgment —is the core function of democratic life..
- Kay Rubacek, America’s Real Crisis: The Collapse Of The Citizen
Terry Winograd was the graduate advisor to Larry Page and Sergei Brin, the founders of Google. Winograd suggested the topic of internet search to them. The book I’m quoting from caused quite a stir in the AI community when it was published in 1987.
The essence of intelligence is to act appropriately when there is no simple pre-definition of the problem or the space of states in which to search for a solution. Rational search within a problem space is not possible until the space itself has been created, and is useful only to the extent that the formal structure corresponds effectively to the situation.
It should be no surprise, then, that the area in which artificial intelligence has had the the greatest difficulty is in the programming of common sense...We accuse people of lacking common sense precisely when some representation of the situation has blinded them to a space of potentially relevant actions.
- Terry Winograd and Fernando Flores, Understanding Computers and Cognition