2.3 ‘Black Box’ AI and Article 36 Reviews
As noted in my previous letter, Article 36 Reviews are conducted to ensure that a weapon is capable of being used lawfully, having regard to all of the principles of IHL, in the context of its use case. The MOD is currently assessing whether the current approach to legal review of weapons requires adjustment for AI-enabled capabilities, working in collaboration with partners to identify international standards.
The potential challenges around explainability of AI-enabled systems may vary depending on the weapons system and its use context. We do have a deep understanding of how machine learning systems are designed and trained and how they make decisions based upon the weights and biases within the network, but it is also true that it can be difficult or impossible to explain an individual decision in human-understandable terms. What matters is the effectiveness and performance of these systems — this is why our Testing, Evaluation, Verification and Validation process (the AI equivalent of medical trials and regulation) is so important. While we cannot explain any individual decision, we can understand the overall level of risk associated with a system and its performance and can therefore make judgements on its effectiveness, safety and the risk-benefit of its use.
We would expect that a future AI-enabled weapon enabled by AI of limited explainability might nevertheless pass an Article 36 Review provided those limitations were overcome by commensurately robust and effective supporting controls and processes. For example, where elements of the technology are less explainable, we may need to tighten the parameters for real-time human oversight or else put restrictions on functionality or the use of the weapon in a certain contexts.
Letter from James Cartlidge MP, the Minister of State for Defence Procurement, to Lord Lisvane, Chair, Lords Artificial Intelligence in Weapons Systems Committee (13 November 2023)
