Algorithmic Accountability Frameworks for High-Risk AI Systems in Public Sector Procurement

Abstract

Government agencies are increasingly procuring artificial intelligence systems to support decisions in welfare eligibility, policing resource allocation, and public benefits administration, yet existing procurement frameworks were not designed to evaluate algorithmic risk. This paper examines how accountability mechanisms can be embedded within public sector procurement processes for systems classified as high risk under emerging regulatory regimes. Drawing on document analysis of procurement guidelines from several jurisdictions alongside semi-structured interviews with procurement officers and vendor compliance staff, we identify recurring gaps between stated transparency requirements and the technical documentation vendors actually provide at the tendering stage. We propose a staged accountability framework that introduces algorithmic impact assessments, independent audit rights, and post-deployment monitoring obligations as binding procurement conditions rather than voluntary vendor disclosures. The framework distinguishes obligations appropriate at pre-award, contract, and operational phases, allowing procurement teams without deep technical expertise to apply consistent evaluation criteria. Findings suggest that procurement, rather than standalone AI legislation, functions as an underused lever for enforcing accountability because it directly controls market access for vendors. We discuss implications for harmonizing procurement-based accountability with broader AI governance frameworks currently under development.