The United Nations Secretary-General António Guterres and the head of the International Committee of the Red Cross and Red Crescent (ICRC) Mirjana Spoljaric Egger issued a joint appeal in August that the world is “dangerously close to crossing a moral red line: the autonomous targeting of humans by machines”.
They said that two types of autonomous weapons — unpredictable autonomous weapons and anti-personnel weapons — must be prohibited.
Mélanie Régimbal, chief of disarmament affairs at the UN reiterated that systems selecting or engaging targets and taking human lives in the absence of control and judgment “are politically unacceptable, morally repugnant and should be prohibited under international law”.
Laurent Gisel, head of the arms and the conduct of hostilities division at the International Committee for the Red Cross (ICRC), described conflict as “not a story of precision, distinction and restraints” but one “of civilians killed, families displaced, hospitals, schools and homes destroyed at ever greater speed and scale”.
In any conflict, many variables come into play, for which autonomous weapons systems cannot be programmed.
“A combatant can surrender or be wounded and therefore be protected from attack; civilians can enter the area of operations of the autonomous weapons systems; the weapons may encounter a situation that would not have been anticipated.”
The dangers of unpredictability in such autonomous systems was hugely problematic. “If it’s not possible to predict it, then how is it possible to limit it,” asked Gisel.
The UN and ICRC issued an appeal on the subject in 2023, calling on countries to negotiate a legally binding document by 2026. The Stop Killer Robots coalition, formed by civil society in October 2012 and launched in 2013, focuses on fostering a world where “we respect each other’s inherent dignity” and take responsibility for choices regarding technology and how it alters the relationships between humans.
It is also concerned with developing technology to promote peace, justice, human rights, equality and respect for law; and where people work as activists “to build a society and overcome inequalities and systems of oppression”.
Its central goal is to develop and fortify “social norms that reject autonomous killing by machines in warfare, policing, border control and other circumstances”. Control, responsibility, accountability are fundamental.
The campaign has not made much meaningful progress, however, given the inexorable rise in global military spending and an increased focus on autonomous warfare and AI.
The 2027 US budget, for instance, calls for US$1.5 trillion in defence spending, with $13.4 billion earmarked for “autonomy and autonomous systems”.
Within the first few weeks of the Gaza war, beginning on October 7, 2023, the Israeli Defense Forces (IDF) made generous use of Lavender, an AI decision-support system (DSS) to generate 37,000 individual targets on a crude rating scale of significance depending on affiliations with armed groups.
Other systems the IDF used include “The Gospel”, which generates target lists, and “Where’s Daddy?”, a platform which tracks the locations of individuals ahead of the strike.
The resulting slaughter of innocents tallied.
The disagreement between the artificial intelligence corporation Anthropic and the US Department of Defense over the dangerous limits of autonomous systems in war is not necessarily a boon for those favouring limits on killer robots or fully autonomous military platforms.
The falling out prompted the Pentagon to blacklist the firm as a “supply-chain risk”. This was found on August 27 by the US District Court of the Northern District of California to be an “unlawful retaliation”, in violation of the First Amendment and a denial of due process as protected by the Fifth Amendment.
Dario Amodei, the Anthropic’s co-founder, conceded in February that the Pentagon “not private companies” made military decisions. “We have never raised objections to particular military operations nor attempted to limit our technology in an ad hoc manner.”
Amodei, however, also claims AI can “undermine” democracy. He cited the company’s mass domestic surveillance technology, which, he said, presented “serious, novel risks to our fundamental liberties” and the use of fully autonomous weapons “that take humans out of the loop entirely and automate selecting and engaging targets”.
But he is not against fully autonomous systems, as they “may prove critical for our national defense”. His concern lies more in the reliability of available systems and whether they would harm American personnel. “We will not knowingly provide a product that puts America’s warfighters and civilians at risk.”
Two days after Amodei made his statement, the US and Israel launched their illegal attack on Iran. Targets were generated at speed by the US military’s use of Palantir’s Maven Smart System, which integrates Anthropic’s Claude AI to assess surveillance data, generate lists of targets and prioritise them.
“Many of the targets hit in Iran have been civilian, including a school and healthcare and residential facilities, illustrating the risks of rapid target generation, wrote Noor Hammad of the International Institute for Strategic Studies.
Gisel told reporters at the end of August that there is no “confirmed evidence that autonomous weapons have been used to directly target human beings in the battlefield”. But given the deferential attitudes to the promise of automated warfare shown by the IDF technology fetishists, the Pentagon and other militaries, it is wise to remain sceptical.
[Binoy Kampmark currently lectures at RMIT University.]