Global Safety Net for AI Riddled with Holes

A sweeping investigation by leading global experts has uncovered significant, and in some cases serious, weaknesses in the very tools used to declare artificial intelligence models safe and effective. The study, released on Monday, November 3, 2025, was conducted by a consortium of computer scientists from the British government’s AI Safety Institute, alongside researchers from prestigious universities including Stanford, Berkeley, and Oxford. They examined over 440 benchmarks—standardized tests used to measure an AI's capabilities and safety—and found that “almost all … have weaknesses in at least one area.”

These flaws, the report states, are severe enough to “undermine the validity of the resulting claims,” meaning that the scores technology companies use to assure the public of their products' safety could be “irrelevant or even misleading.” The investigation comes amid a global surge in AI adoption and growing concerns over harms ranging from character defamation to more severe societal risks. Andrew Bean, the study's lead author from the Oxford Internet Institute, emphasized the gravity of the findings, stating, “Benchmarks underpin nearly all claims about advances in AI. But without shared definitions and sound measurement, it becomes hard to know whether models are genuinely improving or just appearing to.”

Implications for Kenya's 'Silicon Savannah'

These international findings have profound implications for Kenya, a nation rapidly positioning itself as a continental leader in technology and AI. On March 27, 2025, the government launched its ambitious National Artificial Intelligence Strategy (2025–2030), a roadmap designed to make Kenya a regional hub for AI research and responsible innovation. The strategy explicitly prioritizes ethical and responsible AI, aiming to build public trust and ensure that AI systems align with Kenyan values and constitutional safeguards against discrimination.

However, the revelation that the global benchmarks for AI safety are unreliable presents a direct challenge to this vision. As Kenyan businesses and government agencies increasingly procure and deploy AI systems—often developed in the Global North—they may be relying on a false sense of security. These systems are being integrated into critical sectors of the Kenyan economy, including finance for credit scoring, agriculture for crop monitoring, and healthcare for diagnostics. If the safety and effectiveness of these tools cannot be accurately verified, the risk of deploying biased, unsafe, or simply ineffective technology in Kenya grows substantially.

The Risk of Importing Flawed Technology

Kenyan digital rights advocates and policy experts have long warned about the dangers of adopting technology without local context and rigorous oversight. Grace Mutung'u, a prominent Kenyan researcher and advocate for the High Court of Kenya, has extensively researched the impact of digital technologies on human rights in Africa and has highlighted how data collection processes can embed historical biases against women and other marginalized groups. The study's findings amplify these concerns; if the tests themselves are flawed, they are unlikely to detect the nuanced biases that could disproportionately harm Kenyans.

For example, an AI model used for loan applications that was benchmarked with flawed tests might perpetuate biases against certain ethnic groups or genders, undermining financial inclusion efforts. Similarly, AI tools in healthcare trained on non-representative data could lead to misdiagnoses. These risks are not theoretical. A 2024 report by the Mozilla Foundation on AI use among Kenyan creatives found that users already experience significant Western and Eurocentric biases in existing AI tools.

Professor Bitange Ndemo, Kenya's Ambassador to Belgium and the former chairman of the task force that developed the country's digital transformation roadmap, has been a vocal proponent of embracing AI for economic competitiveness. While advocating for adoption, he has also stressed the need for a deep understanding of the technology. Speaking at Davos in February 2025, he urged Africa to “move on with AI and begin to do serious things with it,” framing it as a crucial tool for future growth. The new report underscores that doing “serious things” with AI must include a serious, independent, and contextually relevant verification of its safety and effectiveness, beyond the potentially misleading scores provided by developers.

A Call for Robust Local Governance and Standards

The international study concludes there is a “pressing need for shared standards and best practices” in AI evaluation. This aligns with the goals of Kenya's own National AI Strategy, which calls for the development of a robust governance framework and local talent. The findings should serve as a critical alert for Kenyan policymakers and regulators at the Ministry of Information, Communications, and the Digital Economy.

As Kenya implements its strategy, the focus must now expand to include the development of local or regionally-specific benchmarks and testing protocols that reflect the country's unique linguistic, cultural, and socio-economic realities. The Kenya Bureau of Standards (KEBS) has already initiated public consultation on a Draft Code of Practice for AI, a positive step that could be strengthened by incorporating the lessons from this global report. Without the capacity to independently audit and validate AI models, Kenya risks becoming a passive consumer of potentially flawed technologies, undermining the very goals of safety, inclusivity, and trust that its national strategy aims to achieve. The path forward requires not just adoption, but critical, independent evaluation to ensure AI serves all Kenyans safely and equitably.