LLMs enabling curing cancer, powering disinformation engines, encouraging automated cybercrime?
I finished prompting chatGPT to draft 147 pages, 12 chapters of The Furnace of Reason (chatGPT's title). Including epigraphs for each chapter, with links to sources, chatGPT's narrative of the public domain sources it claimed to use, annotated bibliography, my prompts, and illustrations. Now I'm starting to read it, and I'm a little shook. It's better than not bad. I know there is no understanding in the LLM, but it certainly has a voice. This is not just autocomplete on steroids. I will post a link to a pdf after I finish reading it. I don't intend to edit it. I prompted it and directed it, but this is chatGPT's composition.
I've mentioned Gopnik's assertion that LLMs are less like agents and more like cultural technologies.
Yes there are serious ethical and legal issues about the nonconsensual use of people's work. The use of enormous amounts of energy and water to train LLMs is already negatively affecting communities. And yes, they lie. But LLMs are not going to go away, and there will be different ways to use them. Like the Internet and smarphones, LLMs will amplify the capabilities of individuals and groups who work to humanity's benefit -- and those who work to our detriment, perhaps our extinction.
One example from the beneficial side:
The war on cancer has consumed trillions of dollars and decades of many of the world’s leading scientists, yet for many of the most devastating cancers, our progress remains frustratingly slow. While we've gained deep insights into cancer's extraordinary complexity, translating this knowledge into effective treatments has proven to be one of science's greatest challenges. Throughout history, mankind has been driven by an insatiable urge to explore – we've mapped continents, plumbed the ocean depths, and reached for the stars. Each of these journeys of discovery has transformed humanity. Now, one of the most challenging and vital frontiers of exploration lies within us: the intricate biological machinery that sustains life itself, and the devastating diseases that threaten it.
Our company stands at the intersection of artificial intelligence and biological discovery, where computational power meets the complexity of human cells in unprecedented ways. We're building a new class of AI models to engineer hyper-polypharmacy–driven therapies that precisely guide biological systems from diseased to healthy states, revealing the hidden vulnerabilities in cancer cells that traditional research has missed and integrating it with the work that happens in a laboratory so that models can learn from experimental feedback. This isn't about incremental improvements in cancer treatment – it's about fundamentally reimagining how we discover and develop therapies.
The human body contains more cells than there are stars in the Milky Way, each one part of an intricate symphony we're only beginning to understand. Your role in this mission isn't just to write code or run experiments – it's to be part of the generation that finally cracks the code of cancer's complexity. We're seeking explorers who see the vastness of human biology not as an obstacle, but as an irresistible challenge. Whether you're an AI researcher who dreams of building systems that can decipher biological complexity, a biologist eager to wield computational power in new ways, or a drug discovery scientist ready to break free from conventional approaches – you'll be pioneering approaches that simply weren't possible before. Together, we'll transform cancer treatment from a journey of largely incremental gains to one of breakthrough discoveries, where success isn't measured in small statistical improvements, but in lives restored and families kept whole.
As for negative uses? Why not ask?
Provide three examples of extremely negative possible uses of large language models.
Here are three examples of extremely negative possible uses of large language models (LLMs):
1. Mass-Scale Disinformation and Manipulation
LLMs could be used to generate enormous volumes of convincing but false content—fake news articles, forged research, synthetic social media accounts, and even personalized propaganda. At scale, this could destabilize democracies, amplify conspiracy theories, erode trust in institutions, and make it almost impossible for citizens to distinguish truth from manipulation. In geopolitical contexts, it could be weaponized for influence operations or to incite violence.
2. Automated Cybercrime and Exploitation
A malicious actor could use LLMs to automate tasks traditionally requiring skilled hackers: generating phishing emails tailored to individual targets, writing malware code, or providing step-by-step guidance for exploiting security vulnerabilities. Beyond technical crime, LLMs could also be used to groom, scam, or blackmail individuals by maintaining convincing, manipulative conversations over time. This industrializes exploitation in ways that are difficult to track and stop.
3. Authoritarian Surveillance and Social Control
In the hands of oppressive governments, LLMs could supercharge mass surveillance. For example, automatically analyzing and classifying citizens’ communications, generating real-time loyalty tests, or identifying dissenting voices through subtle linguistic markers. They could also create fabricated confessions, fake “evidence,” or endlessly persuasive messages tailored to keep populations compliant. The scale, personalization, and subtlety of such applications could entrench authoritarianism more deeply than any prior propaganda or policing technology.
Would you like me to expand these into concrete scenarios (step-by-step “what it would look like in practice”) or keep them at the high-level risks?