Whitepaper on algorithmic bias in (Lethal) Autonomous Weapons Systems by Alycia Colijn* and Heramb Podar*
Why do we need to address bias when we speak about (Lethal) Automated Weapons Systems, (L)AWS in short? In this report, we set out which types of bias should be taken into account, when they occur in the lifecycle of AWS, and what this means for policymakers.
Although the rolling text at the moment of writing1 (September 2024) refers to unwanted bias in data sets, and unwanted automation bias, these two types of biases do not cover the full spectrum of bias. To clarify, bias is considered to be any type of flaw in an algorithmic system that leads to a statistic estimate that does not equal the true value2. For example, an unfair over- or underrepresentation of specific groups of people based on gender3, ethnicity, sexual orientation, religion or location, or the misclassification of objects (e.g. hospitals, places of worship, etc.).
| Pre-existing bias | Technical bias | Emergent bias |
|---|---|---|
| The first type of bias we recognize is the pre-existing bias4. This refers to any type of bias that already exists in society and is thus replicated, and often enlarged, by algorithms. This would include biased data sets as included in the rolling text. | Technical bias refers to any kind of bias that occurs from the limitations of a system. This could include a system drawing from an alphabetic list, unintentionally favoring options first up in the alphabetic order. | The last type of bias occurs over time, as a result from changing societal knowledge, population or cultural values. This could also include algorithmic decay, model drift or degradation. |
The first stage of development and deployment of systems is the selection of data that will be used for the training and testing of the system. These data sets are incredibly vulnerable to bias, mostly pre-existing bias. This can originate from the data sets, lack of available data (especially in military use5), but also from the data selection by developers. Where a developer could be held accountable for biased data selection, who is accountable for data sets that reflect a certain societal status quo — including the bias that already exists in society?
When the required data is collected, the model development (also referred to as model training stage) takes off. During this stage, the parameters of the models are fine-tuned. This includes decisions like: will the system make a final decision when it’s 90% sure, 95% sure, or 99% sure? Each parameter that is set comes with the risk of further enlarging bias that already exists in the data that the training stage started with (e.g. class imbalance6 – a preference for bigger ‘classes,’ for example caused by network forming in collaborative filtering algorithms7, and overfitting8 – the model recognizing random noise as a trend) and include (unconscious) bias of the developers.
It is common practice for data scientists to randomly split the initial dataset into two parts: one for training the model (model development) and the other for testing it (model validation), a process referred to as cross validation9. However, when the original data set contains certain bias, the model is then validated with a biased data set as well (and thus not really validated for the ‘real world’).
Additionally, this is the stage where technical bias comes in. To spot bias that is caused by technical limitations, it is vital that developers with different backgrounds analyze the process of model development and evaluation.
Once in use, emerging bias is the greatest risk. This happens when the world changes, and the model is not re-trained. The loss of accuracy can be referred to as degradation10, model drift11, data drift12 or decay12. Data drift, degradation or decay occurs when the data that was used to train (develop) and test (validate) the algorithm no longer reflects the situation in which the model takes decisions, which is sometimes referred to as a distributional shift in environments. In a military context, this for example happens when a system is trained in a specific environment, which changes the longer an armed conflict continues. Model drift includes data drift, but includes other types of drift that lead to a change between the input and output variables, e.g. changing (legal) definitions or changes in military uniforms that challenge the recognition and classification of combatants.
Next to degradation, the self-learning capacities of AI can cause a negative self-reinforcing feedback loop13, which could be considered a form of overfitting over time. The model then identifies noise as a pattern, labeling for example individuals with a certain physical appearance or geographical location as targets.
Automation bias14, as referred to in the rolling text, then occurs when these fallacies are not corrected by human decision-makers because they place a higher degree of trust in the system than in human decision-making.
An unbiased system does not exist. However, there are ways to mitigate these risks as much as possible. For policy-makers, the different risks of bias mean that there are several aspects to take into account when moving to a next step in the discussion around (L)AWS.
ICRC’s Blog Series on AI in the Military
UNIDIR Report on AI and Gender
Whitepaper by Alycia Colijn (The Netherlands) and Heramb Podar (India)
