Computational Quackery
We are in the early days of Large Language Models and so-called “agents”. People are throwing these tools at every conceivable task, right down to being people’s “lovers”. They are being encouraged to do so by loss-leader pricing and the doomster-level FOMO propaganda (”It’s inevitable. Resistance is futile.”) of LLM peddlers.
Corporations are ingesting this technology without the slightest idea of the long-term impact on their organizations.
Obviously there will be some safe and profitable uses of LLMs, but the blanket rollout of this poorly understood technology into every aspect of corporate and personal life is akin to the radioactive quackery of a century ago.
Radioactive quackery is quackery that improperly promotes radioactivity as a therapy for illnesses. Unlike radiotherapy, which is the scientifically sound use of radiation for the destruction of cells (usually cancer cells), quackery pseudo-scientifically promotes involving radioactive substances as a method of healing for cells and tissues. It was most popular during the early 20th century, after the discovery in 1896 of radioactive decay
- Wikipedia, Radioactive Quackery
Before its dangers were known, the highly radioactive element (radium) was pitched as a glowing miracle cure for everything from pimples to cancer.
- History Channel, Radium: The Deadly Health Fad of the Early 1900s
The corporate marketing push
An often overlooked facet of radioactive quackery was the advertising behind it, and the lack of regulations that allowed the outlandish medical claims.
There are many historical summaries of this era. A story in Popular Science recounts two of the more egregious proponents:
1. William Bailey
William Bailey promised to cure anything that ailed you....Between 1924 and 1930, (his list of ailments) list would grow to include more than 150 diseases and discomforts.
(but) Bailey did not create the American craze for radium, he merely joined the rush to capitalize on it [NOTE]...Radium “was mainstream, and it became mainstream because the radium industry wanted this to happen,” explains Maria Rentetzi a historian of science and technology. “Science and commerce are so intertwined that we cannot really separate them,” she says.
- from PopSci Article: The radioactive ‘miracle water’ that killed its believers
2. Eben Byers
The story of Radithor’s best-known victim has also endured: In 1927, Eben Byers , a wealthy and well-known Pittsburgh businessman, broke his arm and a physician recommended Radithor. Over the course of the next five years, Byers swallowed an estimated 2,800 or more ounces of water laced with two radioactive isotopes: radium 226 and radium 228. He died in 1932 of massive radiation poisoning; the Radithor had eaten through his skeleton.
Eben Byers hadn’t been fooled into consuming radium; every bottle of Radithor proudly announced itself as “CERTIFIED Radioactive Water.” Byers had been caught in the intersection where fledgling scientific understanding met an untapped commercial market. The new American radium industry, led by Pittsburgh’s Standard Chemical Company which mined and extracted the element, had a product with promise and a marketing plan that outpaced the scientific process, Rentetzi writes in her book, Seduced by Radium: How Industry Transformed Science in the American Marketplace.
- from PopSci Article: The radioactive ‘miracle water’ that killed its believers
A review of Rentetzi’s book, Seduced by Radium, summarizes the corporate influence:
The discovery of radium by Marie and Pierre Curie in 1898 eventually led to a craze for radium products in the 1920s until their widespread use proved lethal for consumers, patients, and medical practitioners alike. Radium infiltrated American culture, Maria Rentetzi reveals, not only because of its potential to treat cancer but because it was transformed from a scientific object into a familiar, desirable commodity. She explores how Standard Chemical Company in Canonsburg, Pennsylvania—the first successful commercial producer of radium in the United States—aggressively promoted the benefits of radium therapy and its curative properties as part of a lucrative business strategy....Seduced by Radium ultimately reveals how innovative advertising techniques and seductive, state-of-the-art packaging made radium a routine part of American life, shaping scientific knowledge about it and the identities of those who consumed it.
- Publisher’s summary for Seduced by Radium
The corporate cancer cases are starting to trickle in
This year, the two major peddlers of LLMs, OpenAI and Anthropic, began to sell their products by usage (per token) instead of by heavily subsidized flat rate subscription. The financial impact of something approaching the true cost of LLMs has corporations frantically backpeddling from “tokenmaxing”. They are instituting accounting controls and procedures for validating that applications have measurable value.
Beyond the financial impact, corporations are starting to see the negative impact of “AI slop” on both their productivity and their organizational coherence.
Companies that went all-in on using generative AI tools are developing a major “workslop” problem.
In their pursuit to boost productivity, become less reliant on human labor, and reassure investors that they’re riding the cutting edge of tech, some nagging issues are cropping up. As the Harvard Business Review points out, over-relying on AI can prove disastrous for organizational knowledge, the critical business insights companies need to make strategic decisions.
The phenomenon, dubbed “knowledge decay,” describes the deterioration of information over time, marked by workers forgetting skills and organizations relying on outdated processes. In the context of AI, it can be a dangerous downward spiral that starts with workers using AI to produce low-quality work, which wastes colleagues’ time, erodes trust, and gradually sloppifies organizational knowledge into worthless soup.
Multiply that by entire departments, and a business’s outputs start to crumble as well, as the Harvard Business Review explains.
- Futurism, Companies that embraced AI are rotting away
You just have to look at the wording (decay, worthless soup, crumble, downward spiral) to see that there are strong similarities between the way indiscriminent and inappropriate use of AI rots organizations and the way radium rotted patent medicine consumers.
Here is a relevant example of the mechanism of the rot directly from the HBR story mentioned by Futurism:
The third challenge is entropy—the gradual decline of systems into disorder. As knowledge is increasingly passed through AI, it moves further away from the original content used to create it.
For example, we spoke to a healthcare provider who receives long legal documents from insurance firms. They know these are generated with AI. And because they are sent in large numbers, they are being reviewed by AI as human review is cost-prohibitive. What ensues is a risky AI-based game of telephone, as the information with each cycle will increasingly depart from the underlying ground truth knowledge.
This degradation is a function of how the transformer algorithms that underpin all LLMs work. The greater the number of iterations of content through an LLM, the more it will depart from the original. Entropy can be managed, but not eradicated, as long as generative AI models use this underlying technology.
As much as LLMs can create an illusion of thinking and even reasoning, they are probabilistic models providing next-word-prediction style output....New tools and approaches will improve the quality of output, but unless there is a fundamental step change in the architecture of our models, this problem will not go away.
An even more serious version of this problem is known as “model collapse,” which occurs when LLMs are trained on synthetic data, i.e., data created by another LLM or a previous version of the same one. This problem, also known as “generative inbreeding,” eventually can affect the LLM’s accuracy and variability. The causes and impacts of this issue are technically complex, but it is another reason to try to preserve as much human-created content as possible. Studies suggest that already as much as half of the content on the internet and social media is AI-generated, and invariably will become training data for future AI models. Paradoxically, preventing knowledge decay is as important for those that provide AI systems as for those using them.
- Matthias Holweg and Thomas H. Davenport, Don’t Let AI Slop Muck Up Your Company’s Processes
And what do the gentlemen from HBR recommend?
Restrict the use of generative AI.
Companies should only allow the use of the technology where it really adds value
Define what value is being added.
Leaders must be clear what value has been provided by using generative AI, and how. Content does not need to be entirely human-created, but if AI is being used, be clear why and how.
For many tasks, using public LLMs often adds little to no real value. It creates generic prose that often contains mistakes. But the use of proprietary models and/or leveraging proprietary data may well add value.
Understand the implications for the entire process.
...they don’t assess how individual usage can impact an entire process in terms of productivity and output quality....Increasing numbers of companies feel that they need to redesign their business processes to get the full benefit from AI, and as they do so they should consider how to preserve the integrity of content across the process.
What’s the bottom line?
As a technically trained person, I am insulted by the level of propaganda used to sell this product. Like radiation, the technology will turn out to have some valuable use cases. But deskilling the entire population by encouraging them to use LLMs to write college papers, legal briefs, or medical diagnoses will result in the kind of invisible long-term harm that such a deskilled populace will be unable to notice until the economy is seizing up from workslop and model collapse.
I believe the same thing that the Harvard Business Review does. Massive, blind adoption of this technology is bad in the short term and fatal to the economy and the society in the long run. There is a dire need for effective regulation of this dangerous technology.
[NOTE] Oak Ridge National Labs has an interesting collection of radioactive quack products of the era.