AI Offers Poisons, People Make Decisions
AI Offers Poisons, People Make Decisions
An essay on poisons to ploughshares, or the duality of medicines.
AI Poisons
Last year, researchers from Collaboration Pharmaceuticals ran an experiment; in a red team-style effort to assess “how AI techniques for drug discovery could potentially be misused,” they took artificial-intelligence-powered drug design software—conventionally used to minimize drug toxicity—and tampered with the code. Specifically, they tweaked it so that, rather than spit out molecules with minimal toxicities, the model would offer its most potent poisons.
The results, published last month in Nature Machine Intelligence, were striking. In just six hours, the model generated tens of thousands of molecules predicted to have potent activity as chemical weapons. Some of these were known molecules, such as VX nerve agent, while others were entirely new structures with no resemblance to any known chemical weapon—but they were predicted to be even more toxic. “By inverting the use of our machine learning models,” the authors wrote, “we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.”
Well, maybe.
Uncertain Claims
The prospect of new, potent nerve agents is certainly concerning. (After all, the old ones were impressive enough to carry plotlines in pop culture vehicles like The Rock, Dexter, and Homeland.) Still, we should be careful about overreacting where chemicals such as these are concerned. For one thing, drug design is a complicated, multi-step process that comes with a serious set of challenges—stability, volatility, raw materials and intermediates, (industrial) manufacturing capacity and regulatory oversight over every step of the process. Those roadblocks partly explain why AI alone hasn’t been enough to bless us with miracle remedies. It probably won’t be enough to curse us with apocalyptic poisons, either. At least, not in the near term.
But what about the more distant future? What happens when an unscrupulous scientist finally succeeds in willing one of these chemical maledictions into being? Would we be right to panic then? If the lengthy history of these things serves as any indication, then the blessings and curses of chemical biology are not always so clearly distinguished from one another.
Of course, none of that stops the media, from Vice to The Verge to The Financial Times, from picking up the story and running with doomsday headlines. The new study “should raise alarm bells,” one contributor wrote. “If a bad faith actor were to use these models to design highly potent chemical weapons, or, worse, use them, it could be unlike any chemical attack we have seen before,” another worried. “All it takes is one nation prepared to flout international norms, perhaps one led by an autocrat dedicated to malevolence,” wrote a third. And “it would not be surprising if such projects were already underway.”
The hyperbole of these conversations reflect the general lack of nuance and measure which is, unfortunately, so characteristic of the current era. As history forever seems ready to teach us, however, our conversations were not always this way.
The Cloudy History of Chemical Weapons
Writers have explored the duality of chemicals since at least classical antiquity. Homer mentions poison-tipped arrows in both The Iliad and The Odyssey, and Socrates famously died from drinking hemlock. Still, the Greeks—who were no strangers to ambiguous blessings and curses uttered, such as those uttered by the Oracle of Delphi—had a more subtle and sophisticated understanding of poisons than we do today.
“The Greeks understood the two-faced nature of drugs, an understanding reflected in the ambiguity of their term for them: pharmakon,” writes Michael Pollan in This is Your Mind on Plants. “A pharmakon can be either a medicine or a poison; it all depends—on the use, dose, intention and set and setting.”
That understanding did not die with the Greeks. Paracelsus, the 16th century father of modern toxicology, further codified the idea for future generations. “All things are a poison and nothing is without poison,” he taught. “Only the dose makes a thing not a poison.” (In today’s classrooms, this is usually shortened to “the dose makes the poison.”) What he meant was that even the most benign of molecules—water, maybe, or oxygen—could be lethal in certain doses. Shakespeare, who placed poisons at the center of several of his tragedies, wrote clearly on this point:
Within the infant rind of this small flower
Poison hath residence and medicine power:
For this, being smelt, with that part cheers each part;
Being tasted, slays all senses with the heart.
Two such opposed kings encamp them still
In man as well as herbs, grace and rude will;
And where the worser is predominant,
Full soon the canker death eats up that plant.
Several centuries later, World War I introduced a similar poison to the small flowers of Flanders fields. The “chemist’s war,” as it was called even in its own time, marked the first widespread use of poison as a means of chemical warfare. Combatant nations employed scientists like Fritz Haber to synthesize highly lethal compounds, such as chlorine, phosgene and mustard gas, and then weaponized those compounds to cause more than a million gas-related casualties.
The debate surrounding chemical weapons was complicated. J.B.S. Haldane wrote favorably of them, and the American Chemical Society (ACS)—one of the largest scientific societies still in existence today—repeatedly urged its members to lobby in defense of chemical warfare. Some chemists defended these poisons on the grounds that they could be used to incapacitate rather than kill enemy combatants. For his part, however, Fritz Haber did not bother with these sorts of justifications. “Death is death,” he said concisely, “no matter how it is inflicted.”
In contrast to the pleas and complaints of chemists, the soldiers who actually inhaled these poison gases (and lived to tell about it) decidedly condemned them. Gone were the fragile, two-faced flowers of Pollan’s Greek pharmakon or Shakespeare’s Friar Laurence; in their stead, the artists of the Great War left behind a generation of morbid paintings, memoirs, and poetry, their language unsubtle in the extreme. Occult, mystical, and at times supernatural, eye witness accounts described “the loneliness of an accursed, tragic masquerade,” and lamented the denizens of gaseous battlefields as “lost souls expiating some horrible sin in a new Inferno.” Often, soldiers felt that they had descended into Hell itself. “Each time I blundered into sentries or troops who had lost their way,” one soldier explained, “I had the icy sensation of conversing not with people but with demons.”
And yet, like Shakespeare’s flower, even a weapon like mustard gas held medicinal power. Because when researchers studied exposed soldiers, they found that the poison had suppressed their bone marrow and reduced their white blood cell counts. This made it a promising candidate for treating certain blood cancers, like leukemias and lymphomas. So even though mustard gas has cursed hundreds of thousands of people, it has blessed hundreds of thousands of others. (Haber himself has as complicated a legacy: while his chemical weapons destroyed lives—including those of his suicidal wife and, later, child—his work on ammonia fertilizers has probably saved billions from starvation.)
To return to the present day, the authors of the Nature Machine Intelligence paper feared that their study had transformed their AI-powered drug discovery model “from a helpful tool of medicine to a generator of likely dead molecules.” But what, in the end, is a poison? “It’s a substance that interferes with normal physiological processes, that alters or stops them, or makes things happen,” explains biologist Mark Siddall, the former curator of the American Museum of Natural History. “That is essentially what medicines are, too.” Helpful tool of medicine? Generator of likely deadly molecules? Should we be so sure that there’s a difference? After all, today’s cancer chemotherapies were once hidden within clouds of mustard gas. How many of tomorrow’s medicines might be sitting among all those scary nerve agents at Collaborations Pharmaceuticals?
From Poisons to Ploughshares
Chemical weapons are not the only destructive military inventions that were later repurposed for constructive, civilian ends. Government navigation systems, for instance, gave rise over time to now-ubiquitous GPS technologies, and the weapons-grade uranium that powered Soviet weapons has been modified and used for commercial energy. In fact, this practice predates Paracelsus and Shakespeare, and mentions of it in print appear as early as the Old Testament. “They shall beat their swords into ploughshares, and their spears into pruning hooks,” reads the Book of Isaiah. “Nation shall not lift up sword against nation, neither shall they learn war any more.”
But this is a fickle calculus: the number of those helped versus harmed by toxic chemicals, satellite navigational systems, and enriched uranium, among others, are not final tallies but running totals. The same straightforward math that counts them as blessings today might—God forbid—count them as curses tomorrow. Are we really supposed to reduce our moral judgment of these molecules to inconstant, elementary school addition problems?
And that assumes we can run the numbers at all. As the philosopher Daniel Dennett points out in his book Darwin’s Dangerous Idea: Evolution and the Meanings of Life, the main failure of this sort of chemical weapons utilitarianism is that moral accounting is fundamentally impossible in the first place:
It is unlikely in the extreme that there could be a feasible algorithm for the sort of global cost-benefit analysis that utilitarianism…requires. Why? Because of what we might call the Three Mile Island Effect. Was the meltdown at the nuclear plant at Three Mile Island a good thing to have happened or a bad thing? […]
How could Three Mile Island have been a good thing? By being the near catastrophe that sounded the alarm that led us away from paths that would encounter much worse adventures—Chernobyl, for instance.
So maybe the prospective chemical weapons, described in the Nature Machine Intelligence paper, will do more good than bad. But we’ll also never know, since the timelines are long and the moral sums are impossible to calculate. Ultimately, drugs with basement-level median lethal doses are really just diverse arrangements of atoms. Our desire to map moral values onto indifferent gases and metals may make for attention-grabbing headlines, but it also threatens to obscure the actual moral center of each of the many examples discussed above: humans, and the ways that they use the tools that they’ve made.
AI can offer new poisons, but we decide how to use them.
About the Author
Chris Bassil is an MD/PhD candidate who uses functional genomics to study drug resistance in cancer.
I think the real important point they were trying to make is not the fact that it's bad that particular AI can produce potentially more powerful weapons, but the fact that dual use examples like this could be widespread, where a research group creates an AI model and others can relatively easily tune it and use it with malevolent intent. That's what should concern us.