Written by Chandrani Mondal, an LL.M. student specialising in AI and Law at Symbiosis Law School, Pune, and Sunaina Mondal, final-year BBA LL.B. student at Symbiosis Law School, Noida. Both have previously interned in Data Privacy at Ernst & Young (EY India), Gurugram.

In 2016, an American man named Eric Loomis received a prison sentence of six years. The sentence was partly based on an algorithm owned by the private company that predicted a high probability of Mr. Loomis reoffending. The man contested the sentencing, arguing that not being able to scrutinize the methodology behind the algorithm deprived him of his due process rights. The Wisconsin Supreme Court dismissed his arguments. According to State v. Loomis, 881 N.W.2d 749 (Wis. 2016), the algorithm was just one piece of information out of several factors. The U.S. Supreme Court refused to entertain his appeal.

It is not important about Loomis’s case occurring in Wisconsin that made it notable. What is important is the fact that Loomis-like situations are emerging all around the world. In the UK, for instance, artificial intelligence systems have been utilized for bail hearings. Sentencing has already been automated in the US. Automated decision-making has also been used for case management in courts of law in China, and for welfare and immigration decisions in the EU and elsewhere.

According to this point of view, the unquestioning dependence of the judiciary on artificial intelligence is a gross breach of basic international human rights principles, namely, the right to a fair trial and equality before the law, which, along with the lack of an international legal regime regulating artificial intelligence in the field of justice, can be considered one of the gravest governance mistakes in our times. Fragmented local solutions, regardless of how elaborate, are not enough anymore. Artificial intelligence in legal processes requires a global solution.

II. Background

A fair trial has always been one of the most universal human rights enshrined in international law. The Article 14 of the International Covenant on Civil and Political Rights (ICCPR), which has been ratified by 173 nations, provides for equality before the courts and tribunals, the right to a public hearing, and the right to be heard regarding the case presented against the person. 

These laws were made at a time when there were no algorithms. No one ever considered the idea that any evidence against a criminal would be produced by something that could not be questioned because its algorithm was a trade secret, because it thought in ways no one else understood, and because no one could call it into court as a witness. The question becomes, do they rule out such an idea when properly interpreted? This approach says they do but international organizations have been far too slow and quiet about it.

Meanwhile, the technology has not been idle. China’s smart court initiative employs algorithmic technology in handling millions of cases each year. India, Kenya, and South Africa have experimented with the application of artificial intelligence in judicial administration and judicial risk assessment. INTERPOL has employed predictive analytics software in its cross-border intelligence efforts against crimes. The EU AI Act which came into force in 2024, is so far the most complete regional regulation, categorizing criminal justice use of AI as high-risk. On the global scale, UNESCO Recommendation on the Ethics of Artificial Intelligence adopted in 2021 by 193 countries is the closest to an international guideline, albeit it lacks enforcement power. 

III. The Argument

1. AI Adjudication as a Violation of International Human Rights Norms

In Loomis, the defense was denied access to the methodology behind the COMPAS algorithm on the basis that it was a trade secret. The accused was convicted based on an input for which he had no access, no interrogation rights, and no capacity to challenge. It is an infringement on the principle of Article 14 of the ICCPR, and not simply a technical problem with procedure. Just because there was technically a human judge involved does not change that fact.

The judgment of the Court of Appeal of England and Wales in R (Bridges) v. Chief Constable of South Wales [2020] EWCA Civ 1058, holding that the use by South Wales Police of live facial recognition technology breached both the right to privacy and equality laws, demonstrates that some domestic courts are starting to seriously consider human rights criteria when applying algorithms in state operations. The importance of the Bridges case is its requirement that adherence to domestic data protection laws is insufficient to fulfill the State’s human rights obligations. International human rights law requires even more.

2. The Jurisdictional Vacuum

AI algorithms that have been designed and tested in one location are used in another. The risk-assessment systems created in the US have influenced debates about AI in Europe, Australia, and throughout the Global South. The facial recognition technology designed by companies in the US, China, and Israel is adopted by police forces in Africa, Asia, and Latin America, often in the absence of any regulatory regime governing its use in those countries and without recourse to the company designing the software. The UN Special Rapporteur on the Right to Privacy has noted that surveillance and predictive technologies have spread across borders without any accountability systems attached to their export, licensing, or use.

Even when there is a need for transparency and accountability as stipulated in the UNESCO Recommendation on AI Ethics one cannot compel countries to abide by it since it is not an international treaty. It does not create any obligations that can be enforced by any court. When a country uses its power against soft laws, the soft laws do not have much effect. The soft laws are like guidelines which do not have to be followed by countries. Countries can choose to ignore them if they want to.

3. The Global South and the Export of Algorithmic Injustice

Machine Bias investigation conducted by ProPublica showed how COMPAS algorithm made predictions about recidivism that were racially skewed in favor of whites and against blacks, overestimating the risks for black people and underestimating those for whites. In a 2019 study by Starre Vartan demonstrated how a healthcare algorithm discriminated against black patients through basing its decisions upon costs of treatment, which were determined by discrimination as a measure of their needs. The logic of both failures, at the structural level, is transferrable: training on data that reflected historically skewed social patterns, so as to produce output that reifies the skew, is an approach, and approaches transfer. 

This was a point made by the UN Special Rapporteur on Extreme Poverty and Human Rights when issuing the 2019 report on Digital welfare states. According to the report, the use of automated decision-making technologies poses a danger because such technology threatens to perpetuate the status quo while undermining the accountability framework necessary for the functioning of both the welfare and legal states. 

SOURCE: Author designed recommendation chart [Visualized by AI]

IV. Conclusion

The basic endeavor of international law is to discipline power with principle. Use of AI in adjudication is an area of burgeoning state power and an area that defines liberty, dictates the lives of many generations and transverses states and jurisdictions with a disregard for all legal principles that seek to frame its usage. The norms of international human rights law already contain all that is needed to address this use – transparency, accountability, the right to a fair trial and equal treatment before the law without discrimination are principles that provide grounds to resist it. 

The court-room has been a space of reason. It becomes something quite different when a non-sentient black box begins authoring judicial reason, unexamined, unchallengeable and unbound by any principle of international law, when instead of reason, only its imitation is rendered. International law needs to do that, which domestic courts have largely been loath to do – hold the machine to account and insist it to justify itself, in terms comprehensible to a defendant and verifiable by a judge, and then defer; or be left to do so itself.