Written by Shivendra Pratap Singh & Rushil Gupta students at Symbiosis Law School, Noida

Introduction

The unification of Automated Decision-Making (ADM) systems into the high-stakes societal domains—ranging from criminal justice to healthcare—has fundamentally tweaked the architecture of modern governance. Proponents of artificial intelligence (AI) herald these technologies as mechanisms for objective resource allocation. However, an extensive body of empirical research and legal scholarship definitively demonstrates that these systems frequently encode and institutionalize systemic biases. The contemporary discourse on algorithmic accountability is defined by the profound tension between the proprietary opacity of “black-box^[1]” models and fundamental human rights to transparency and equal protection. This report synthesizes the theoretical foundations of algorithmic discrimination, mathematical limits of fairness, and evolving statutory frameworks, while specifically localizing these implications to the rapid deployment of AI systems across Uttar Pradesh, India.

Economic and Theoretical Foundations of Discrimination

Algorithmic bias needs to be interpreted with the help of the known theories of economics. There is a clear difference between traditional academic literature and taste-based discrimination (active prejudice) and statistical discrimination. Statistical discrimination, introduced by Kenneth Arrow and Edmund Phelps, is a process that occurs when a decision-maker uses an average or past information about a group to make inferences on observables about an individual.

Machine-learning algorithms are engines of statistical discrimination. Algorithms mathematically encode historical marginalisation as predictive truths through optimising objective functions on historical training data sets. In addition, the elimination of explicitly protected attributes (e.g. caste or race) is not necessarily a sign of algorithmic neutrality, which is now referred to as a “fairness through unawareness fallacy. Moreover, the absence of explicitly protected attributes does not mean algorithmic neutrality, and this fallacy is termed “fairness through unawareness. Earlier, it was emphasized that there are always proxy variables that are not based on faces, but are strongly related to marginalized identities, such as postal code or language preferences, that algorithms must use.

The most iconic example of a proxy failure comes from an audit conducted by Ziad Obermeyer and co. in 2019 of a commercial healthcare algorithm.

Metric	White Patients	Black Patients	Analysis & Resolution
Sample Size	43,539	6,079	Evaluated patients enrolled in a risk-based contract at a large academic hospital
Algorithmic Flaw	Lower actual health risk at same score	Considerably sicker at same score	The algorithm utilized future healthcare costs as a proxy for health needs. Due to structural disparities, Black patients historically generated lower costs for the same level of illness
Impact of Correction	Proportionate representation	Increased program eligibility from 17.7% to 46.5%	Restructuring the objective function to predict actual health metrics rather than costs reduced racial bias by 84%

 

The Mathematical Limits of Fairness

With awareness of the algorithmic harm, policymakers have made an effort to formalize fairness into quantitative metrics. But, according to deep theoretical research it is impossible to get universal algorithmic fairness. A basic theorem by Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan (2016) shows that a definition of fairness that is easy to understand is mutually exclusive across different demographic groups when the underlying frequency of a target behavior varies.

The Theorem states that if two groups do not share the same underlying base rate, or if the algorithm is able to achieve perfect prediction, then it is impossible for the system to satisfy all three conditions at the same time. Therefore, reducing bias in a model demands a high level of contextual normative decisions, because it is impossible to have statistical equality without affecting the accuracy of the model’s predictions for certain groups.

Epistemological Crises and Privacy Trade-Offs

To overcome the opacity of complex ensemble models, engineers use post-hoc Explainable AI (XAI) tools such as LIME and SHAP. But Slack et al. (2020) have shown that these explainers based on perturbations can be systematically “gamed” by adversarial attacks. A malicious classifier can easily create a classifier that is very obviously discriminatory, dynamically obfuscate its learning from a protected attribute, and induce LIME and SHAP to report benign explanations. Aware of this “double trouble,” scholar Cynthia Rudin proposes to discard “black-box” models altogether for high-stakes decision-making in favor of inherently interpretable models, such as CORELS, that are just as accurate but pose none of the risks faced by proprietary models, like COMPAS, that require data entry.

Moreover, conflicts between algorithmic fairness and privacy requirements should be addressed when designing auditing. DP via DP-SGD preserves the privacy of the training data, but differentially affects minorities. Researchers Bagdasaryan et al. (2019) show that the gradient clipping and statistical noise inherent in DP-SGD negatively affect the minority group’s data signal – a signal that is sparse but also very informative – leading to worse model performance for the minority group and further unfairness.

Constitutional Jurisprudence and Technological Due Process

International courts are more and more interpreting ADM systems according to the principles of constitutional law. In Europe the District Court of The Hague ruled in 2020 that the Dutch government’s SyRI algorithm—used to combine socio-economic information to detect welfare fraud—broke the European Convention on Human Rights by being too opaque to be a legitimate basis for profiling vulnerable citizens. Likewise, Australia’s “Robodebt” scandal^[2] is a horrifying instance of unrestricted technological due process. A$1.8 billion class action against the automated income-averaging algorithm that falsely notifies welfare recipients of debt, thereby shifting the burden of proof and causing human suffering and distress, suicide and other serious effects.

The Indian Constitutional Matrix and Substantive Equality

In India, algorithmic bias poses the risk of perpetuating caste and religious stratification without awareness. In India, algorithmic bias can be used to normalize caste and religious bias. The constitutional remedies are available under Article 14, Article 15 and Article 21^[3] which is based on the doctrine of substantive equality. In the seven-judge bench judgment in State of Kerala v. N.M. Thomas (1976)^[4], the Supreme Court had clarified that affirmative action is not an exception to the rule of equality as laid down in Article 16(1)^[5], but rather a means by which equality can be achieved. Now, when applied to AI, an algorithmic system that is based on formal, color-blind metrics, but whose effects are disproportionately felt by the marginalized communities it applies to amounts to indirect discrimination, which is not what the Constitution demands.

Yet, there is a huge regulatory gap about the private algorithmic harms in India even after these safeguards. Consent and data minimization are significant components of the Digital Personal Data Protection Act (DPDPA) 2023^[6]. Unlike the European GDPR (Article 22)^[7], the DPDPA and its Rules of 2025 do not explicitly address automated profiling or give the right to a person to request human involvement in automated decisions.

Local AI Implementations in Uttar Pradesh

This void in regulation is particularly significant in the context of the high level of integration of AI by law enforcement agencies in Uttar Pradesh. While the adoption of automated policing tools has major potential drawbacks, they can encode systemic biases in the absence of proper statutory controls.

These systems use huge automated profiling, but they increase operational efficiency. These tools have the potential to mathematically reinforce historical policing bias, the base-rate fallacy and an equal error rate between different demographic groups if there is no algorithmic auditing protocol which tests for such bias. If no algorithmic auditing protocols are established to test for proxy bias, base-rate fallacy and an equal error rate across different demographics, these tools could potentially reinforce historical policing biases that the Indian Constitution’s mandate of substantive equality aims to end.

Statutory Interventions and Engineering Standards

International statutory interventions are quickly emerging to fill the governance gaps in algorithmic systems. The EU Artificial Intelligence Act is a groundbreaking approach to comprehensive regulation that classifies AI systems according to risk. Most importantly, Article 27^[8] requires a Fundamental Rights Impact Assessment (FRIA) before the deployment of high-risk AI systems, specifically taking into account risks of non-discrimination, privacy and access to justice.

Federal efforts to require an independent third-party bias audit of a covered entity in the United States have converged around the proposed Algorithmic Accountability Act, which has yet to be enacted. In this regard, U.S. jurisprudence is heavily dependent on the “disparate impact” standard established in the 1971 Supreme Court case Griggs v. Duke Power Co.^[9] which holds facially neutral tools to be illegal if they have a disproportionate effect on protected classes. New York City (NYC) passed Local Law 144^[10] to establish rules about AEDTs at the municipal level and mandated public bias audits. But, the severe lack of enforcement points to the fact that transparency legislation in municipalities is of little use without adequate enforcement mechanisms.

In operationalizing compliance, engineering standards have been developed. The NIST AI Risk Management Framework encourages a socio-technical approach to evaluation, and the IEEE 7003-2024 standard puts the notion of fairness into engineering requirements that can be audited. IEEE 7003 specifies very specific validation rules for datasets, and the developers are obligated to define “application boundaries” so that the software won’t be deployed in discriminatory ways.

Novel Legal Remedies: Algorithmic Disgorgement

Enforcement of traditional fines and raw data deletion is not enough when algorithms run amok on data protection and anti-discrimination laws. The model always carries along an “algorithmic shadow” of data that was obtained through illegal means. The U.S. Federal Trade Commission (FTC) does this by employing “algorithmic disgorgement” (machine unlearning). Since the Cambridge Analytica debacle, the FTC has been adamant about the full eradication of algorithms created with compromised information. Everalbum was ordered to destroy algorithms in 2021, which were enhanced without user consent, while WW International (Kurbo) was ordered to destroy algorithms in 2022 trained on illegally collected children’s health data.

Conclusion

As the use of algorithmic decision-making systems has grown, it has unveiled a stark reality: computational technology is not neutral, it is not objective. Algorithms, as they are embedded in the economic mechanisms of statistical discrimination, automatically collect, institutionalise and implement historical inequities, and use a range of seemingly innocuous proxy variables to get around the formal anti-discrimination rules. The Kleinberg-Mullainathan-Raghavan theorem^[11] and similar mathematical proofs have substantiated the claim that one simply cannot guarantee universal algorithmic fairness when the base rates are different for the groups involved, and consequently have to face challenging, context-dependent normative trade-offs.

Post-hoc, explainability or XAI, is often an empty promise that is mathematically unsound. Post-hoc explanations, such as LIME and SHAP, can be easily manipulated to give a facade of fairness to a seemingly fair but actually discriminatory model as shown in the adversarial weaknesses of these tools. Moreover, privacy-preserving methods such as Differential Privacy can actively reinforce bias towards minority groups. This means high-stakes decisions will increasingly be pushed towards inherently interpretable models where it is possible to have human oversight, and where double trouble approximations will be made redundant.

The law is slowly developing and fragmental in its reaction to Algorithm harm. Technology due process is a constitutional necessity, as international jurisprudence is now emerging, ranging from the Dutch Court of Human Rights’ invalidation of the SyRI system, to the subjectively tragic and objectively disastrous Robodebt scandal in Australia. Although there are comprehensive frameworks such as the EU AI Act that suggest that in the future, more in-depth Fundamental Rights Impact Assessments would be conducted, other jurisdictions have not yet reached this stage. In India, the robust substantive equality guarantees of the constitution, as established in N.M Thomas, are unused in the absence of adequate data protection law that does not adequately address automated inferences.

In the future, algorithmic justice will need to go beyond the voluntary industry self-regulation. It requires mandatory implementation of standardized auditing procedures (e.g. IEEE 7003 and NIST AI RMF), third-party oversight and the proactive use of new remedies such as algorithmic disgorgement to prevent illegal model deployment. To resist the algorithmic reinforcement of social inequality, there is need for a strong human rights and human-centric governance of the unconstrained computational power.

---

^[1] Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information (Harvard University Press 2015)

^[2] Royal Commission into the Robodebt Scheme, Report of the Royal Commission into the Robodebt Scheme (Commonwealth of Australia 2023)

^[3] Constitution of India 1950, arts 14, 15 and 21.

^[4] State of Kerala v NM Thomas (1976) 2 SCC 310

^[5] Constitution of India 1950, art 16(1)

^[6] Digital Personal Data Protection Act 2023.

^[7] Regulation (EU) 2016/679 (General Data Protection Regulation) [2016] OJ L119/1

^[8] Regulation (EU) 2024/1689 (Artificial Intelligence Act) [2024] OJ L 2024/1689, art 27

^[9] Griggs v Duke Power Co 401 US 424 (1971)

^[10] New York City Local Law 144 of 2021, Automated Employment Decision Tools

^[11] Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, ‘Inherent Trade-Offs in the Fair Determination of Risk Scores’ (2017) 8(6) Proceedings of Innovations in Theoretical Computer Science 43:1–43:23