When Justice is Blind to Algorithms: Multilayered Blackboxing of Algorithmic Decision Making in the Public Sector

Both research and public discourse have recently drawn attention to the downsides of algorithmic decision making (ADM), highlighting how it can produce biased and discriminatory outcomes, and also pose threats to social justice. We address such threats that emanate from, but also go beyond, algorithms per se, extending to how public agencies and legal institutions (fail to) respond to consequences of ADM. Drawing on a case study of the use of an ADM system in public school administration, we explore the practices through which public institutions avoided engagement with the detrimental consequences of ADM, leading to injustice. We provide a conceptual model outlining how organizational ignoring practices can lead to social and institutional blackboxing of an ADM system, engendering both social and legal injustice. Our work paves the way for interdisciplinary research on multilayered blackboxing of ADM. We also extend algorithmic injustice research to include a legal dimension and provide practical implications in the form of a legal framework for ADM in the public sector.