ICRC IHL Challenges Report (2015)

1) New technologies of warfare

[…]

In accordance with Article 36 of Additional Protocol I, each State Party is required to determine whether the employment of a new weapon, means or method of warfare that it studies, develops, acquires or adopts would, in some or all circumstances, be prohibited by international law. Legal reviews of new weapons, including new technologies of warfare, are a critical measure for States to ensure respect for IHL. More specifically, they are a way to ensure that a State’s armed forces are capable of conducting hostilities in accordance with its international obligations, and that new weapons are not employed prematurely under conditions in which respect for IHL cannot be guaranteed. However, despite this legal requirement and the large number of States that develop or acquire new weapon systems every year, only a small number are known to have procedures in place to carry out legal reviews of new weapons.⁴¹

[…]

The importance of the legal review of cyber capabilities

These and other issues underscore why it is important that States that may develop or acquire cyber-warfare capacities, whether for offensive or defensive purposes, assess their lawfulness under IHL. Legal review, as specifically required by Article 36 of Additional Protocol I, is essential to ensuring that armed forces and other government agencies that may potentially resort to cyber operations in an armed conflict are able to abide by their obligations under international law. However, the legal review of cyber weapons, means and methods of warfare may present a number of challenges. Military cyber capabilities might be less standardized than kinetic weapons, especially if designed for a specific operation; furthermore, they are likely to be subject to constant adaptation, including to respond to the software security upgrades that a potential target will undergo.

In sum, the ICRC believes that clarifying how IHL applies to cyber warfare would help shed light on whether its rules are sufficiently clear in view of the specific characteristics and foreseeable humanitarian impact of cyber warfare. Given that this type of warfare poses novel questions, there may also be a need to develop IHL as technologies evolve or as the human cost of cyber warfare becomes better understood.

[…]

Legal review of autonomous weapon systems

The above challenges will need to be carefully considered by States when carrying out legal reviews of any autonomous weapon system they develop or acquire, as required by IHL. As with all weapons, the lawfulness of a weapon with autonomy in its critical functions depends on its specific characteristics, and whether, given those characteristics, it can be employed in conformity with the rules of IHL in all of the circumstances in which it is intended and expected to be used. The ability to carry out such a review entails fully understanding the weapon’s capabilities and foreseeing its effects, notably through testing. Yet foreseeing such effects will become increasingly difficult if autonomous weapon systems are increasingly able to determine their own actions in complex environments.

Predictability about the actions of an autonomous weapon system in the context in which it is to be deployed must be sufficiently high to allow an accurate legal review. Indeed, deploying a weapon system whose effects are wholly or partially unpredictable would create a significant risk that IHL will not be respected. In this regard, a key question for the reviewer is how to evaluate and mitigate the risks of using the weapon if its performance is unpredictable. The risks may be too high to allow the weapon’s use; otherwise, mitigating the risks may require appropriate levels of human control over the critical functions of the weapon system, consequently limiting or even obviating the weapon’s autonomy.

An additional challenge for reviewing the legality of an autonomous weapon system is the absence of standard methods and protocols for testing these weapons. This too may affect the accuracy of the legal review.

Accountability for the use of autonomous weapon systems

[…]

Under the law of State responsibility, in addition to accountability for violations of IHL committed by its armed forces, a State could be held liable for violations of IHL caused by an autonomous weapon system that it has not, or has inadequately, tested or reviewed prior to deployment. Under the laws of product liability, manufacturers and programmers could also be held accountable for errors in programming or for the malfunction of an autonomous weapon system.

ICRC, International Humanitarian Law and the Challenges of Contemporary Armed Conflicts: Report (October 2015) 38, 43–44, 46