Start with the Problem, Not the Tech
Before procuring any AI solution, the first step is to clearly define the problem it's meant to solve. Many AI projects fail not because the technology is flawed, but because they are solutions in search of a problem. Instead of asking, 'How can we use AI?,'
agencies should ask, 'What is the most pressing challenge we face, and could AI help solve it effectively?' This approach, often called 'challenge-focused procurement,' encourages a focus on outcomes rather than specific technologies. Clearly outlining the desired public benefit and objectives provides a benchmark against which all potential AI tools can be measured, ensuring the technology serves a genuine public need, whether it's reducing administrative burdens, improving citizen inquiry response times, or optimising resource allocation.
Prioritise Data Quality and Governance
AI systems are only as good as the data they are trained on. The principle of 'garbage in, garbage out' is especially critical in the public sector, where datasets can contain sensitive citizen information. Before even considering a vendor, an institution must assess its own data landscape. Is the data clean, complete, and representative of the entire population it serves? Historical data can reflect societal biases, and if used without care, an AI tool can perpetuate or even amplify discrimination. Establishing strong data governance is non-negotiable. This involves creating clear policies on data collection, storage, and access, ensuring that privacy and security are paramount from the outset. A thorough data assessment is a prerequisite for any responsible AI procurement process.
Scrutinise for Fairness and Bias
One of the most significant risks of using AI in government is the potential for algorithmic bias. An AI system used for social service planning or tax fraud detection could unfairly target certain demographic groups if its training data is skewed. Therefore, a rigorous assessment for fairness is crucial. Public institutions should require vendors to be transparent about how their models were trained and what steps they have taken to mitigate bias. This includes reviewing test reports and vetting the vendor's bias mitigation strategies. It’s also vital to involve diverse stakeholders in the design and review process to identify potential blind spots that a purely technical team might miss. The goal is to ensure the AI tool treats all citizens equitably and does not embed or deepen existing inequalities.
Demand Transparency and Explainability
For public institutions, decisions cannot be made inside an unexplainable 'black box'. If a citizen is denied a benefit or flagged by an AI system, the agency must be able to explain why. This is a core tenet of public accountability. When assessing AI tools, agencies should prioritise systems that offer transparency and explainability. This means the vendor must be able to provide clear documentation on the AI's logic and how it arrives at its conclusions. While full transparency of a complex algorithm might be difficult, the ability for a human to trace and justify an AI-assisted decision is non-negotiable. This ensures that a human remains in control and accountable for the final outcome.
Conduct Rigorous Pilot Programs
An AI tool that performs well in a lab environment may behave differently in the real world. Before a full-scale rollout, conducting a pilot program is an essential risk mitigation step. A pilot allows the institution to test the AI solution in a controlled, real-world context and measure its performance against predefined benchmarks and key performance indicators (KPIs). This is the stage to evaluate the tool's true impact on citizen services, identify any unintended consequences, and gather feedback from both employees and the public. It provides an opportunity to iterate and refine the system before it affects the entire population, ensuring the technology is not only effective but also safe and reliable for public use.
Plan for the Entire Lifecycle
Procuring an AI tool is not a one-time purchase; it is the beginning of a long-term commitment. The assessment process must consider the entire lifecycle of the system. This includes evaluating the vendor’s provisions for ongoing support, maintenance, and training. What happens when the model needs to be updated? Who is responsible for continuous monitoring to ensure performance and fairness don't degrade over time? A robust plan should also include clear accountability mechanisms, such as audit rights and remediation procedures, to manage the system responsibly throughout its operational life. This long-term view ensures that the AI tool remains a valuable and trustworthy asset for the institution and the citizens it serves.
