Article: Judges-in-the-loop? Judicial involvement in human oversight of high-risk decision support systems under the EU AI Act

Abstract

INTRODUCTION

The new European Union (EU) Artificial Intelligence Act (AI Act) makes human oversight central to the governance of high-risk AI systems.¹ According to Article 14 of the AI Act, high-risk AI systems must be ‘designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use’.² The purpose of human oversight under the AI Act is to ‘prevent or minimise the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used’—in particular those that persist despite the application of the regulation’s other high-risk system requirements.³

The need for humans to remain in control of AI systems used in the judicial context specifically is affirmed by Recital 61 of the AI Act, which states: ‘[t]he use of AI tools can support the decision-making power of judges or judicial independence, but should not replace it: the final decision-making must remain a human-driven activity’. AI systems that are used to assist judicial authorities are ostensibly among those understood to pose ‘a significant risk of harm to the health, safety, or fundamental rights of natural persons’⁴ and are therefore the focus of this article’s discussion of judicial involvement in human oversight. It is important to note, however, that the vast majority of judicial decision support systems are likely to be exempted from the regulation’s high-risk classification.⁵ Article 6(3) includes a derogation for AI systems that do ‘not materially influenc[e] the outcome of decision making’, including those intended to: ‘perform a narrow procedural task’, ‘improve the result of a previously completed human activity’, ‘detect decision-making patterns or deviations from prior decision-making patterns and [not] replace or influence the previously completed human assessment, without proper human review’, or ‘perform a preparatory task to an assessment relevant for the purposes of the use cases listed in Annex III’.⁶ Only systems that ‘perform profiling⁷ of natural persons’ will always be considered high-risk and therefore subject to the AI Act’s human oversight requirement, irrespective of the degree of human involvement in the decision-making process.⁸

Judicial decision support systems that perform profiling of natural persons typically profile litigants. These systems use machine learning or statistical models to generate outputs (eg risk scores or categories), predictions, or recommendations about litigants to inform judicial decision-making.⁹ In the criminal justice context, profiling systems typically take the form of risk assessment tools designed to estimate the likelihood that a defendant will commit a crime in the future or will not return to court if released from jail pretrial.¹⁰ Examples include COMPAS in the USA, HART in the UK, and OxRec in Sweden and the Netherlands. Profiling systems can also be used to support judicial decision-making in civil or administrative contexts. Examples include the Mexican Experitus system, which was used to advise judges and clerks about litigants’ pension eligibility, or SyRI, which was used by the Dutch Ministry of Social Affairs and Employment to detect various forms of benefits and tax fraud.¹¹

Against this background, this article explores the implications of making judges who use high-risk decision support systems (those that perform profiling) responsible for their oversight under Article 14 of the AI Act. Specifically, it assesses judges’ capacity to fulfil the expectations of individuals tasked with human oversight set out in Article 14(4) in light of their unique professional responsibilities and empirical findings about the ways they interact with high-risk decision support systems in practice. Judges who use decision support systems have counterfactual influence over the systems’ outputs: they are in a position to assess and override them if they choose. This means that to the extent they are made responsible for human oversight under the AI Act—and to the extent they perform this oversight while using a high-risk decision support system—they can be considered ‘in-the-loop’ of the AI-supported decision-making process.¹² Although judicial users of high-risk decision support systems are not the only actors who could potentially be made responsible for their oversight under Article 14, the requirements of judicial independence and the expectation that human oversight take place ‘during the period in which [the AI system is] in use’¹³ arguably makes them logical candidates for this role. This article therefore takes ‘judges-in-the-loop’ as the starting point for its discussion of the implications of making judges responsible for human oversight under the AI Act. It concludes by considering who else could be involved in human oversight of high-risk judicial decision support systems, and what other forms judicial involvement in this oversight might take.

‘Human oversight under the AI Act’ introduces the AI Act’s human oversight requirements and explains what is legally expected of providers of high-risk AI systems, the institutions that deploy them, and the ‘natural persons’ to whom human oversight is assigned. ‘Professional responsibilities of judges who use high-risk decision support systems’ draws from national, international, and assistive technology-specific codes of ethical judicial conduct to identify the values that judicial users of high-risk decision support systems are professionally obligated to uphold, and discusses how they relate to the expectations of individuals tasked with human oversight under the AI Act. ‘How judges interpret and exercise discretion over high-risk decision support systems in practice’ presents empirical research revealing how judges interact with high-risk decision support systems in real-world settings and again discusses how these findings relate to the expectations of individuals tasked with human oversight under the AI Act.

‘Judges-in-the-loophole’ discusses what the professional responsibilities, skills, motivations, and biases of judges surfaced in the previous two sections suggest about their capacity to perform human-in-the-loop oversight of high-risk decision support systems under the AI Act, and draws attention to the accountability gap that expecting them to do so could create. Finally, ‘Rethinking judicial involvement in human oversight of high-risk decision support systems under the AI Act’ makes the case for an alternative form of judicial involvement in human oversight of high-risk decision support systems that not only meets the AI Act’s requirements but also goes further towards safeguarding fundamental rights and judicial values.

This interdisciplinary analysis draws from a growing body of legal, social science, and system safety research suggesting that human–machine collaboration is difficult to calibrate in practice and often has unintended effects.¹⁴ As Crootof et al. explain, ‘[r]ather than marrying the best of humans and machines, hybrid human-machine systems can exacerbate the worst of each, while adding new sources of error’.¹⁵ Much of the literature highlighting these errors calls for further research exploring how human oversight of AI-supported decision-making operates in specific contexts.¹⁶ This article aims to fill that gap by interpreting the AI Act’s broad, horizontal human oversight obligations in the high-stakes context of AI-supported judicial decision-making, and assessing the implications of making judicial users of profiling systems responsible for carrying them out.

HUMAN OVERSIGHT UNDER THE AI ACT

Read more at 

https://academic.oup.com/ijlit/article/doi/10.1093/ijlit/eaag001/8475419?searchresult=1&login=false