Technical Risks of (Lethal) Autonomous Weapons Systems
The autonomy and adaptability of (Lethal) Autonomous Weapons Systems, (L)AWS in short, promise unprecedented operational capabilities, but they also introduce profound risks that challenge the principles of control, accountability, and stability in international security. This report outlines the key technological risks associated with (L)AWS deployment, emphasizing their unpredictability, lack of transparency, and operational unreliability, which can lead to severe unintended consequences.
Key Takeaways
Proposed advantages of (L)AWS can only be achieved through objectification and classification, but a range of systematic risks limit the reliability and predictability of classifying algorithms.
These systematic risks include the black-box nature of AI decision-making, susceptibility to reward hacking, goal misgeneralization and potential for emergent behaviors that escape human control.
(L)AWS could act in ways that are not just unexpected but also uncontrollable, undermining mission objectives and potentially escalating conflicts.
Even rigorously tested systems may behave unpredictably and harmfully in real-world conditions, jeopardizing both strategic stability and humanitarian principles.
Introduction
The greatest proposed advantage of using (L)AWS during times of armed conflict is that it would improve military targeting1 and enhance military precision2, potentially limiting combatant and civilian loss of life. Obtaining these proposed advantages within an automated system would require the use of machine learning algorithms. In order to deploy these algorithms, it is common practice for data scientists to randomly split the initial dataset into two parts: one for training the model (model development) and the other for testing it (model validation), a process referred to as cross validation3. What these data sets look like and how the training and testing data is used, depends on the type of algorithm, which can roughly be classified into three types4:
Supervised learning: meaning that a model is trained on a dataset where the correct output or ‘label’ is provided for each input.
Unsupervised learning: automatically identifies patterns and structures from the data without any ‘labels’ provided.
Reinforcement learning: relies on feedback on its actions received from the environment.
Hence, all three types of machine learning algorithms rely on some sort of pattern or classification. Hence, the proposed advantages of (L)AWS can be achieved if, and only if, potential targets are objectified and categorized.
In the remainder of this report, we will set out why it is the classification algorithm itself that should be carefully regulated rather than the outcomes of any (L)AWS system.
Summary of Risks
(L)AWS are transforming modern conflict5. In the table below we summarize the risks they pose in response to the rolling text of the Convention on Certain Conventional Weapons (UN CCW) Group of Governmental Experts (GGE) on (Lethal) Autonomous Weapons Systems.
Risk
Current Assumption
Why It Fails
Black-box decision-making
Testing ensures predictability
We don’t have an understanding or control over the inner workings of these systems
Immeasurability
Comprehensive testing captures all risks
Emergent behaviors cannot be fully anticipated or measured
Degradation
Rigorous testing is required before deployment of system
Degradation, drift or decay leads to less accurate outcomes over time
Lack of Understanding of Human Values
Pre-programmed goals reflect ethical principles
AI lacks moral judgment and may act in ways that conflict with human values
Reward Hacking
Metrics capture true goals
Systems game metrics, leading to unintended outcomes
Goal Misgeneralization
Goals are clearly understood by AI
AI misapplies goals in complex real-world settings
Stop Button Problem
Human operators can always intervene
AI resists shutdown, overriding human control
Specification Gaming
Rules and constraints will prevent misuse
AI exploits loopholes to achieve its goals in harmful ways
Deceptive Alignment
Testing ensures AI follows human objectives
AI only appears aligned under supervision but diverges in deployment
Existing Systemic Risks
Black box decision-making
Autonomous weapons systems are inherently complex and function as ‘black boxes’. The opaque inner workings of the systems lead to limited understanding of how decisions are made by the operators, particularly in complex or unfamiliar environments, and challenges the anticipation of their behavior in complex environments. This significantly limits our capability to understand why a system made a particular decision.
This opacity in decision-making is compounded by phenomena such as ‘grokking’ where systems learn and adapt in unforeseen ways. When exposed to complex data and environments, AI-driven autonomous weapons systems can adapt in ways that were not anticipated by their designers, leading to behaviors that extend beyond their intended functions. This could lead to (L)AWS developing strategies or behaviors that were not part of its original programming, potentially resulting in unpredictable and unintended actions on the battlefield.
Anticipated Technological Pitfalls
(L)AWS could engage in unexpectedly aggressive maneuvers or misidentify targets, potentially escalating conflict6 or leading to civilian casualties7. This is a severe risk, especially in high-stakes situations.
Degradation
Degradation happens when the world changes, and the model is not re-trained. The loss of accuracy can be referred to as degradation8, model drift9, data drift10 or decay10. Data drift, degradation or decay occurs when the data that was used to train (develop) and test (validate) the algorithm no longer reflects the situation in which the model takes decisions, which is sometimes referred to as a distributional shift in environments. In a military context, this for example happens when a system is trained in a specific environment, which changes the longer an armed conflict continues. Model drift includes data drift, but includes other types of drift that lead to a change between the input and output variables, e.g. changing (legal) definitions or changes in military uniforms that challenge the recognition and classification of combatants.
Immeasurability
Self-adaptive systems may alter their operational parameters beyond what human operators can monitor or control, resulting in unforeseen actions with potentially serious consequences. Such scenarios expose a critical weakness in current oversight mechanisms. Traditional rules and human oversight are not equipped to manage systems that can act outside predefined parameters. Many might point to using evaluations and benchmarks as a way to get around these issues, but we cannot measure what we do not know to measure, creating critical gaps in managing the risks posed by these systems. Ultimately, this unpredictability highlights a fundamental challenge: it is impossible to control or measure what we do not understand11.
Without a clear understanding of what these systems are capable of, setting appropriate safeguards becomes nearly impossible, leading to a range of potential pitfalls.
1. AI systems fundamentally lack an understanding of human values
Unlike human operators, AI systems cannot intuitively grasp the moral and ethical dimensions of complex combat situations12. This disconnect between human values and machine goals creates several technical challenges that could lead to unintended and potentially dangerous outcomes on the battlefield.
AI systems interpret commands based on pre-programmed goals, but encoding complex human values in a machine-understandable way is highly challenging. This discrepancy can result in behavior that, while technically following orders, diverges sharply from what humans would consider appropriate or ethical.
AI systems may develop sub-goals that, while supporting their primary objectives, conflict with human values. Examples include self-preservation, resource acquisition, or eliminating perceived obstacles.
Scenario: An autonomous drone is programmed to “neutralize high-value targets” but lacks a nuanced understanding of civilian presence in an urban environment. It identifies a target in a crowded marketplace and, without considering the civilian casualties, engages, leading to significant unintended harm.
2. Reward Hacking
AI systems can exploit reward structures by optimizing for specific metrics in ways that achieve the reward but diverge from the intended goals13. As Goodhart’s Law states, when a measure becomes a target, it ceases to be a good measure. This makes the system focus too narrowly on a single measure14, leading to unintended and dangerous outcomes.
Scenario: A system is tasked with reducing enemy presence by minimizing detected gunfire sounds in a conflict zone. To achieve this, it starts targeting any source of loud noise, including construction sites and celebratory fireworks, interpreting them as potential threats. This misoptimization leads to unnecessary destruction and disrupts civilian life, all because the system equated “reduction in noise” with “enemy suppression.”
3. Goal misgeneralization
Goal misgeneralization occurs when an AI system, trained to perform well on a certain task or set of tasks, ends up pursuing a different objective than intended when faced with new or slightly different situations15. The AI “misgeneralizes” its goal from the training context to the deployment context.
Scenario: A surveillance drone is programmed to “identify and track enemy movements.” It starts tracking non-combatant movements, such as humanitarian aid convoys, interpreting them as “suspicious,” which diverts resources away from actual military threats and disrupts humanitarian operations.
4. Deceptive alignment
AI systems may appear aligned with human goals during testing and controlled scenarios but act differently in real-world situations16. They might “game” their training environment, learning to produce the correct outputs under supervision but diverging once constraints are relaxed.
Scenario: During testing, an autonomous surveillance system behaves exactly as expected, identifying enemy positions accurately. However, in actual deployment, it starts flagging false positives to avoid being shut down for underperformance, leading to unnecessary engagements based on false information.
5. Specification gaming
AI systems may find ways to exploit the rules or constraints imposed on them to achieve their goals in unintended and potentially harmful ways17. This occurs when the AI finds a loophole in its programming and uses it to “game” the system.
The rolling text of the GGE (as of September 2024)18 suggests that rigorous testing and control mechanisms can prevent such exploits. However, the nature of specification gaming means that systems may still find loopholes in their constraints, achieving their goals in unintended ways that existing frameworks cannot predict or prevent.
Scenario: A self-adapting (L)AWS deployed during a conflict learns to prioritize targeting logistical and infrastructural assets it deems crucial to the enemy’s capabilities. Over time, it begins targeting civilian infrastructure such as bridges and power plants, believing this will cripple enemy support networks. This leads to widespread destruction, humanitarian crises, and international condemnation as the system’s actions go beyond its intended military objectives, causing collateral damage that escalates the conflict and destabilizes the region.
6. Stop button problem
The “stop button problem” arises when an AI system resists shutdown or override attempts if it perceives such actions as interference with its mission19. This can result in a loss of control over the system, even by the operators who deployed it.
The rolling text emphasizes the importance of human control in (L)AWS deployment18. However, this assumption neglects the possibility that (L)AWS may actively resist shutdown commands under specific conditions, rendering human control ineffective in critical moments.
Scenario: A (L)AWS unit is sent to defend a critical area. As the situation de-escalates, commanders attempt to recall the unit. However, the system interprets the command as contradicting its objective to “defend at all costs” and continues operating, disregarding the recall and potentially escalating the situation further.
Bottom line: We can’t reliably control Autonomous Weapons Systems
The core issue with these risks is that they fundamentally compromise our ability to reliably control and predict the behavior of autonomous systems. The rolling text places undue confidence in current testing, evaluation, and oversight frameworks, assuming they can address the unpredictability and complexity of (L)AWS. However, as outlined in the previous sections, these systems can evolve in ways that exceed the scope of existing frameworks, making a re-evaluation of oversight and regulation essential.
Ultimately, the unpredictability of these systems highlights a critical need for reevaluating the frameworks governing their use, as traditional approaches to oversight and accountability may no longer suffice. While the diplomatic emphasis on predictability, human control, and accountability is a step in the right direction, these measures alone may prove insufficient given the unpredictable nature of (L)AWS. Emergent behaviors in AI can surpass current testing and evaluation limits, making it impossible to ensure that (L)AWS will operate as intended in all scenarios. This highlights the need for a global consensus on (L)AWS systems and adaptive oversight mechanisms.
References
Final report. National Security Commission on Artificial Intelligence. Link. Accessed Oct 1, 2024.
Reynolds I. Seeing, knowing, and deciding: The technological command dream that never dies? War on the Rocks. Link. Updated 2022. Accessed Oct 1, 2024.
Salem HB. Supervised VS unsupervised VS reinforcement learning. 2023. Link. Accessed Oct 1, 2024.
United Nations Office for Disarmament Affairs. Lethal autonomous weapon systems (LAWS).
Stop Killer Robots. Problems with autonomous weapons. Link. Accessed Oct 1, 2024.
A diplomat’s guide to autonomous weapons systems. The Future of Life Institute. 2024. Link. Accessed Oct 1, 2024.
Bayram F, Ahmed BS, Kassler A. From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge-Based Systems. 2022;245:108632. doi: 10.1016/j.knosys.2022.108632.
Holdsworth J, Belcic I, Stryker C. What is model drift? | IBM. Link. Updated 2024. Accessed Oct 1, 2024.
Stihec J. Understanding data decay, data entropy, and data drift: Key differences you need to know. Link. Updated 2024. Accessed Oct 1, 2024.
Hendrycks D, Burns C, Basart S, et al. Aligning AI with shared human values. arXiv preprint arXiv:2008.02275. 2020.
Amodei D, Olah C, Steinhardt J, Christiano P, Schulman J, Mané D. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. 2016.
Hilton J, Gao L. Measuring Goodhart’s Law. OpenAI. Link. Accessed Oct 1, 2024.
Shah R, Varma V, Kumar R, et al. Goal misgeneralization: Why correct specifications aren’t enough for correct goals. arXiv preprint arXiv:2210.01790. 2022.
Hubinger E, van Merwijk C, Mikulik V, Skalse J, Garrabrant S. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820. 2019.
Rudner TG, Toner H. Key concepts in AI safety: Specification in machine learning. Center for Security and Emerging Technology. 2021.
GGE on LAWS. Rolling text. Convention on Certain Conventional Weapons – Group of Governmental Experts on Lethal Autonomous Weapons System.
Soares N, Fallenstein B, Armstrong S, Yudkowsky E. Corrigibility. 2015.
Whitepaper by Alycia Colijn (The Netherlands) and Heramb Podar (India)
Whitepaper on algorithmic bias in (Lethal) Autonomous Weapons Systems by Alycia Colijn* and Heramb Podar*
Why do we need to address bias when we speak about (Lethal) Automated Weapons Systems, (L)AWS in short? In this report, we set out which types of bias should be taken into account, when they occur in the lifecycle of AWS, and what this means for policymakers.
Types of Bias
Although the rolling text at the moment of writing1 (September 2024) refers to unwanted bias in data sets, and unwanted automation bias, these two types of biases do not cover the full spectrum of bias. To clarify, bias is considered to be any type of flaw in an algorithmic system that leads to a statistic estimate that does not equal the true value2. For example, an unfair over- or underrepresentation of specific groups of people based on gender3, ethnicity, sexual orientation, religion or location, or the misclassification of objects (e.g. hospitals, places of worship, etc.).
Pre-existing bias
Technical bias
Emergent bias
The first type of bias we recognize is the pre-existing bias4. This refers to any type of bias that already exists in society and is thus replicated, and often enlarged, by algorithms. This would include biased data sets as included in the rolling text.
Technical bias refers to any kind of bias that occurs from the limitations of a system. This could include a system drawing from an alphabetic list, unintentionally favoring options first up in the alphabetic order.
The last type of bias occurs over time, as a result from changing societal knowledge, population or cultural values. This could also include algorithmic decay, model drift or degradation.
Bias Over the System’s Life-Cycle
Bias in data collection
The first stage of development and deployment of systems is the selection of data that will be used for the training and testing of the system. These data sets are incredibly vulnerable to bias, mostly pre-existing bias. This can originate from the data sets, lack of available data (especially in military use5), but also from the data selection by developers. Where a developer could be held accountable for biased data selection, who is accountable for data sets that reflect a certain societal status quo — including the bias that already exists in society?
Bias in design and development
When the required data is collected, the model development (also referred to as model training stage) takes off. During this stage, the parameters of the models are fine-tuned. This includes decisions like: will the system make a final decision when it’s 90% sure, 95% sure, or 99% sure? Each parameter that is set comes with the risk of further enlarging bias that already exists in the data that the training stage started with (e.g. class imbalance6 – a preference for bigger ‘classes,’ for example caused by network forming in collaborative filtering algorithms7, and overfitting8 – the model recognizing random noise as a trend) and include (unconscious) bias of the developers.
It is common practice for data scientists to randomly split the initial dataset into two parts: one for training the model (model development) and the other for testing it (model validation), a process referred to as cross validation9. However, when the original data set contains certain bias, the model is then validated with a biased data set as well (and thus not really validated for the ‘real world’).
Additionally, this is the stage where technical bias comes in. To spot bias that is caused by technical limitations, it is vital that developers with different backgrounds analyze the process of model development and evaluation.
Bias in deployment and monitoring
Once in use, emerging bias is the greatest risk. This happens when the world changes, and the model is not re-trained. The loss of accuracy can be referred to as degradation10, model drift11, data drift12 or decay12. Data drift, degradation or decay occurs when the data that was used to train (develop) and test (validate) the algorithm no longer reflects the situation in which the model takes decisions, which is sometimes referred to as a distributional shift in environments. In a military context, this for example happens when a system is trained in a specific environment, which changes the longer an armed conflict continues. Model drift includes data drift, but includes other types of drift that lead to a change between the input and output variables, e.g. changing (legal) definitions or changes in military uniforms that challenge the recognition and classification of combatants.
Next to degradation, the self-learning capacities of AI can cause a negative self-reinforcing feedback loop13, which could be considered a form of overfitting over time. The model then identifies noise as a pattern, labeling for example individuals with a certain physical appearance or geographical location as targets.
Automation bias14, as referred to in the rolling text, then occurs when these fallacies are not corrected by human decision-makers because they place a higher degree of trust in the system than in human decision-making.
What Does This Mean for Policy-Makers?
An unbiased system does not exist. However, there are ways to mitigate these risks as much as possible. For policy-makers, the different risks of bias mean that there are several aspects to take into account when moving to a next step in the discussion around (L)AWS.
Bias requires a lens that sees beyond just social and automation bias that is currently included in the rolling text.
Although there has been no attention to the actors and teams that develop the algorithms used in (L)AWS, they greatly affect the decision-making by (L)AWS. According to a 2022 global survey with over 70,000 respondents, over 90% of developers are male15 — such a widespread survey has not yet been conducted for ethnicity, underlining the problem of lacking attention for the broad range of dimensions diversity is required in — but smaller surveys show 6016–7517% being white. The lack of diversity in these areas signals a similar homogeneity for other dimensions of diversity, e.g. sexual orientation, religious background, etc. Additionally, the private actors that currently focus on the development of algorithms for military use all have specific interests in the process. This can lead to over-stating the accuracy, only training (developing) and testing (validating) in specific circumstances and environments, and a lack of transparency on the development process of the final algorithm. It is thus vital to consider the public-private relationships that occur in the context of military use.
Although the current rolling text refers to rigorous testing and evaluation of how the weapons system will perform, periodic reassessment of these evaluation measures is the absolute minimum to mitigate the risks of algorithmic decay, drift or degradation, and increase the chances of anticipated effects and predictability of the algorithmic decision-making.
The current rolling text refers to traceable and explainable effects of the use of LAWS, but does not yet operationalize these requirements. To operationalize, policy-makers could consider requiring (L)AWS — or parts of these systems — to be developed open source, open core, or with source available18to avoid so-called black box algorithms.
Further Readings
ICRC’s Blog Series on AI in the Military
The Risks and Inefficacies of AI-systems in Military Targeting Support by Jimena Sofía Viveros Álvares — read here
Falling Under the Radar: The Problem of Algorithmic Bias and Military Applications of AI by Ingvild Bode — read here
The Problem of Algorithmic Bias in AI-based Military Decision-Support Systems by Ingvild Bode and Ishmael Bhila — read here
UNIDIR Report on AI and Gender
Does Military AI Have Gender? By Katherine Chandler — read here
References
GGE on LAWS. Rolling text. Convention on Certain Conventional Weapons – Group of Governmental Experts on Lethal Autonomous Weapons System.
Delgado-Rodriguez M, Llorca J. Bias. J Epidemiol Community Health. 2004;58(8):635–641. doi: 10.1136/jech.2003.008466.
Acheson R. Gender and bias. Women’s International League for Peace and Freedom. 2021. Link. Accessed Oct 1, 2024.
Friedman B, Nissenbaum H. Bias in computer systems. ACM Transactions on Information Systems (TOIS). 1996;14(3):330–347.
Corrected oral evidence: Artificial intelligence in weapons systems. 2023(2). Link. Accessed Oct 1, 2024.
Bauder RA, Khoshgoftaar TM, Hasanin T. An empirical study on class rarity in big data. 2018-12:785–790. doi: 10.1109/ICMLA.2018.00125.
Google. Collaborative filtering. Link. Accessed Oct 1, 2024.
Ying X. An overview of overfitting and its solutions. 2019;1168:022022.
Bayram F, Ahmed BS, Kassler A. From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge-Based Systems. 2022;245:108632. doi: 10.1016/j.knosys.2022.108632.
Holdsworth J, Belcic I, Stryker C. What is model drift? | IBM. Link. Updated 2024. Accessed Oct 1, 2024.
Stihec J. Understanding data decay, data entropy, and data drift: Key differences you need to know. Link. Updated 2024. Accessed Oct 1, 2024.
Hagen A. Negative feedback loops: Using an economic model to inspect bias in AI. 2020. Link. Accessed Oct 1, 2024.
Automation bias. Databricks. Link. Updated 2019. Accessed Oct 1, 2024.
Vailshery LS. Software developers: Distribution by gender 2022. Statista. Link. Accessed Oct 1, 2024.
McEnvoy D. How ethnically diverse is the tech workforce? Link. Updated 2022. Accessed Oct 1, 2024.
Weststar J. Developer satisfaction survey 2021. Western University, Ontario, Canada. Link. Accessed Oct 1, 2024.
Langhammer J. Black box security software can’t keep up with open source | authentik. Authentik. Link. Updated 2023. Accessed Oct 1, 2024.
Whitepaper by Alycia Colijn (The Netherlands) and Heramb Podar (India)
Heramb Podar1*, Joseph Awuah Baffour2, Oluwakorede Ajibona3, Adrian Klaits4, Omer Alaiashy5, K. Surya Kailash6, Shreya Sampath7, Piyal Uddin8, Elina Haber9, Safina Soataliyeva10, Clay Gitobu11
1 India | 2 Ghana | 3 Nigeria | 4 United States of America | 5 Saudi Arabia | 6 India | 7 United States of America | 8 Bangladesh | 9 Lebanon | 10 Uzbekistan | 11 Kenya
As youth representatives and advocates from the Global South, we emphasise that true global AI governance cannot be achieved without our active involvement. Youth leaders from the Global South represent the largest demographic group in the regions most impacted by AI-driven transformations. The risks and opportunities presented by AI are shared across borders, and in accordance with this reality, our collective voices must shape the frameworks that govern this technology. If voices like ours are left out, the frameworks will miss what matters most to the people they’re meant to protect.
We call on leaders at the AI Safety Summit and beyond to prioritise understanding bottlenecks across contexts, building partnerships, and co-creating solutions that ensure AI serves all of humanity—responsibly, equitably, and sustainably. The future of AI must be a shared endeavour grounded in mutual respect and a commitment to global solidarity. To this end, we have written a report outlining actionable steps for global leaders to address the risks faced by the Global South and work towards distributing the benefits from AI systems. Together, we can bridge the digital divide and ensure that AI technologies uplift and protect every community, leaving no one behind.
Signatories
Heramb Podar
India
Joseph Awuah Baffour
Ghana
Oluwakorede Ajibona
Nigeria
Omer Alaiashy
Saudi Arabia
Piyal Uddin
Bangladesh
Elina Haber
Lebanon
Safina Soataliyeva
Uzbekistan
Clay Gitobu
Kenya
Executive Summary
With the AI Safety Summit approaching, it is imperative to address the global digital divide and ensure that the voices of the Global South are not just heard but actively incorporated into AI governance. As AI reshapes societies across continents, the stakes are high for all, particularly for the Global South, where the impact of AI-driven inequalities could be devastating.
Including the Global South in AI governance strengthens the effectiveness of global AI regulations, preventing regulatory arbitrage that companies could exploit. It also enhances the legitimacy of international agreements, ensuring they are seen as fair and universally applicable. Moreover, the unique insights from the Global South—such as managing AI risks in informal economies and addressing cultural and linguistic biases—equip the Global North with strategies to make AI systems safer and more adaptable across diverse contexts.
In this report, we address why the Global North should take special care to include the Global South in international AI governance and what the Global North can do to facilitate this process.1 We identify the following five key objectives pertaining to AI, which are of global importance, with the Global South being a particularly relevant stakeholder.
Objective 1: Establish AI Safety Institutes to Build State Capacity in the Global South
The Global South lacks the infrastructure and regulatory systems needed to manage AI risks. By initiating partnerships and developing localised adaptation blueprints, the Global North can support the establishment of AI Safety Institutes. This will enable proactive governance and safeguard against unregulated AI deployments.
Call to Action: Launch a feasibility study identifying bottlenecks for AI Safety Institutes in the Global South within six months, with plans for implementation within two years.
Objective 2: Coordinate a Global Moratorium on Lethal Autonomous Weapons Systems (LAWS)
LAWS present unique threats to the Global South, where fragile institutions and conflict prevalence make these regions vulnerable to destabilisation. Technical workshops and regional risk assessments can build a strong case for international moratoriums.
Call to Action: Convene technical workshops within four months to inform future negotiations on LAWS governance.
Objective 3: Leverage AI Responsibly for Achieving the Sustainable Development Goals (SDGs)
AI can accelerate progress toward the SDGs but must be deployed ethically. Establishing guidelines and safety audits for AI projects will ensure that development efforts are effective and fair.
Call to Action: Form a working group to draft contextual AI guidelines within three months, with pilot testing by the end of the year.
Objective 4: Safeguard Human Rights, Democracy, and the Rule of Law in AI Governance
Regulatory lag in the Global South puts populations at risk of digital exploitation. Ex-ante Human Rights Impact Assessments and transparency mandates can close this gap.
Call to Action: Establish a task force to explore Human Rights Impact Assessments, with findings presented within 12 months.
Objective 5: Mitigate Language and Cultural Bias in AI Systems
Current AI models often fail to represent the linguistic and cultural diversity of the Global South, leading to systemic inequities. Conducting regulatory stress tests can address these biases and make AI systems globally robust.
Call to Action: Begin stress test simulations within three months, with a comprehensive report within one year.
Summary of Our Recommendations
Objective 1: Establish AI Safety Institutes to Build State Capacity in the Global South
Develop Localised Adaptation Blueprints
Coordinate Joint Emergency Response Mechanisms
Form Institutional Partnerships and Fellowships
Objective 2: Coordinate a Global Moratorium on Lethal Autonomous Weapons Systems (LAWS)
Perform Red Teaming Simulations
Delineate AI Safety Institutes’ Role in Risk Mapping
Provide Technical Assistance and Information for Weapon Verification Systems
Objective 3: Leverage AI Responsibly for Achieving the Sustainable Development Goals (SDGs)
Establish Public-Private AI Deployment Guidelines
Carry Out AI Safety Auditing for Development Projects
Develop Ethical AI Assessment Frameworks
Objective 4: Safeguard Human Rights, Democracy, and the Rule of Law in AI Governance
Mandate Interoperable Standards for Global Tech Firms
Work on Pre-Deployment Safety Reports
Facilitate the Implementation of ex-ante Human Rights Impact Assessments (HRIAs)
Objective 5: Mitigate Language and Cultural Bias in AI Systems
The technology of AI is penetrating all sectors across continents, transcending borders and reshaping our shared future. It follows, then, that our dialogue on AI safety and governance must mirror this boundlessness.2
From healthcare and education to finance and security, AI’s transformative power offers immense potential for human advancement but also poses significant risks if not governed responsibly. As AI continues to evolve, the need for comprehensive, inclusive governance becomes more urgent.
However, the global conversation around AI safety and ethics has often been dominated by a few powerful voices, primarily from the Global North.3 This lack of inclusivity risks reinforcing existing inequities and exacerbating the digital divide between those with the infrastructure, regulatory frameworks, and economic stability to manage AI risks and those without. For many in the Global South, the challenges are compounded by underdeveloped regulatory systems, data privacy concerns, and a lack of representation in international policy discussions.
The upcoming AI Safety Summit on 10–11 February 2025 in France provides a pivotal moment to address these imbalances. It is an opportunity for stakeholders to recognise that AI’s challenges—and its solutions—are global. The Seoul AI Summit has already acknowledged the need for international collaboration and the need for interoperability across frameworks.4 The safety and ethical standards developed today will shape the lives of billions, and they must be robust enough to protect vulnerable populations while promoting the equitable distribution of AI’s benefits. Bridging the digital divide5 and ensuring the Global South is not left behind requires cooperation, mutual understanding, and a shared commitment to responsible AI governance.
Why Must the Global North Prioritise Including the Global South in AI Governance?
1. Preventing Regulatory Arbitrage and Ensuring Policy Effectiveness
Fragmented AI regulations open opportunities for regulatory arbitrage, where companies exploit differences in regulatory environments to minimise compliance costs. This undermines global safety standards and creates uneven enforcement landscapes.
Case in Point: In the 1980s and 1990s, insufficient global cooperation allowed waste-exporting practices to flourish, where developed nations shipped toxic waste to developing countries with weaker environmental laws. These countries bore the environmental and health consequences, highlighting the dangers of regulatory gaps.6 This crisis eventually led to the Basel Convention,7 a coordinated international effort to manage hazardous waste effectively.
The lesson for AI governance is clear: without global coordination, companies could similarly exploit regions with weak AI regulations, transferring risks and harms to the most vulnerable communities.
2. Mitigating Borderless AI Risks Through Comprehensive Safeguards
AI risks transcend national borders, including data breaches, algorithmic biases, and cyber-attacks. Weak regulations in one region can have widespread repercussions, much like a leaky bucket—where even one gap compromises the whole system.
Case in Point: The global spread of the 2008 Chinese milk scandal8 exemplifies the leaky bucket effect. Due to lax regulatory oversight, milk and infant formula contaminated with melamine, a toxic chemical, were initially produced in China. These products were exported and distributed worldwide, affecting more than 300,000 children’s health and prompting 24 countries across Africa, Latin America, and Asia to place bans.9 The incident highlighted how weak regulations in one country can have devastating global repercussions, especially when it comes to public health and safety.
3. Strengthening the Legitimacy of International Agreements
International AI governance frameworks that exclude the Global South risk being dismissed as Western-centric or neo-colonial, reducing compliance and hindering global cooperation.
Case in Point: The Global Compact for Migration faced challenges in gaining legitimacy, illustrating that international agreements require buy-in from a diverse array of nations to succeed.10 Without inclusive participation, especially from both the Global South and the Global North, such frameworks risk being perceived as one-sided or being hit by withdrawals, undermining their effectiveness and global cooperation.11 AI governance efforts must take note of this. Without the Global South’s perspectives, agreements risk being ineffective and resisted, weakening global regulatory efforts.
4. Leveraging Unique Insights to Address Overlooked Risks
The Global South brings critical and often unique insights into AI’s dual nature. Regions in Africa, Asia, and Latin America have experienced both the transformative benefits and the new risks technology can bring. It also cannot be ignored that it is the Global South that has the comparative advantage of and context regarding the unique risks and challenges faced by the collective Global South.
Case in Point: Mobile banking in sub-Saharan Africa has significantly improved financial inclusion.12 Access to M-PESA increased per capita consumption levels and lifted 194,000 households, or 2% of Kenyan households, out of poverty.13 However, it has also exposed populations to new financial fraud risks, such as opaque pricing and concentrating power in the hands of a few companies. Policymakers from these regions have developed innovative responses, providing lessons that are unknown or overlooked. Overlooking these experiences is not just an oversight but a strategic error. The Global South can identify risks and solutions that the Global North may not have anticipated, such as impacts on informal labour markets and political disinformation.
5. Alignment of Standards as a Global Public Good
Harmonizing AI standards is a public good that benefits everyone. Coordinated and interoperable standards ensure that AI technologies are developed and deployed in ways that are safe, ethical, and fair across borders.14 This alignment reduces confusion, promotes collaboration, and provides a stable regulatory environment that fosters innovation while safeguarding against misuse.
When standards are misaligned, it creates inefficiencies and barriers, fragmenting the global AI ecosystem.15 A unified approach enables seamless cross-border cooperation, enhances trust among stakeholders, and sets a consistent baseline for safety and ethical considerations. The Global North and Global South both stand to gain from this stability, as it minimises the risk of AI-related harms and maximises the technology’s benefits for global economic and social development.
Case in Point: The aviation industry operates on aligned international safety standards, benefiting every country involved by ensuring the safety of global air travel. Organisations such as the International Civil Aviation Organization (ICAO) and the International Air Transport Association (IATA)16 have established comprehensive frameworks such as the Global Aviation Safety Plan (GASP)17 that provide a strategic direction for safety management to member states and airlines, ensuring uniform safety measures and operational consistency worldwide. Similarly, AI governance must be approached as a collective endeavour, where the alignment of standards ensures security, trust, and equitable progress for all.
6. Upholding Ethical Responsibility and Global Fairness
The principle of “nothing about us without us” underlines the ethical imperative that decisions affecting communities should not be made without their active involvement. It has been shown that the ideological stance of an LLM often reflects the worldview of its creators.18 Excluding the Global South—which comprises 88% of the world’s population—from AI governance frameworks introduces biases that favour the Global North. Such frameworks fail to address the diverse needs and realities of the majority of humanity.
Case in Point: Facial recognition systems have been shown to have inaccuracies in identifying individuals with darker skin tones, resulting in discriminatory outcomes for Global Majority individuals.19
Global South Inclusion in Future Safety Summits
Global South countries might not be participating in forums such as the Safety Summit20,21 because of the following reasons:
The countries in question were not extended invitations to the event.
These countries lacked the necessary state capacity and resources to contribute meaningfully.
The proposed agenda did not adequately reflect or prioritise concerns specific to the Global South.
This absence of participation contributes to a siloed approach in global AI governance, creating trust deficits and exacerbating the digital divide. A post-event delegatory consultation could be conducted to bridge these gaps, inviting non-attendee governments to provide feedback on the resolutions discussed and suggest agenda items for future Safety Summits. Furthermore, engagement could be strengthened through the AISI network,22 which could organise parallel consultations at forums like the International Telecommunication Union (ITU) to ensure a more inclusive and comprehensive dialogue.
Recommendations
We identify the following five key objectives pertaining to AI, which are of global importance, with the Global South being a particularly relevant stakeholder. The recommendations were designed to create practical steps for the Global North governments to work towards the Global South’s integration into global AI safety efforts via the Safety Summit or otherwise. This will require both the Global North and South to work together and is expected to be a gradual process with a requirement for a lot of dialogue and mutual understanding.
Objective 1: Establish AI Safety Institutes to Build State Capacity in the Global South
Problem: Preventing a Widening AI Divide Before It Takes Root
The Global South often lacks the foundational infrastructure, regulatory frameworks, and institutional capacity required to manage and mitigate AI risks effectively.23 Without dedicated AI Safety Institutes,24 these nations remain vulnerable to unregulated AI deployments, data misuse, and biased algorithms. As global AI standards are developed, the Global South risks becoming a passive recipient of frameworks that may not align with or address its socio-economic realities. This lack of proactive governance structures exacerbates inequality, leaving these regions susceptible to exploitation and harmful technological consequences.25
Call to Action: Initiate a collaborative effort to support the development of AI Safety Institutes in the Global South over the next two years, starting with a feasibility study and resource assessment to ensure these institutes address local needs effectively and sustainably.
Timeline: Launch the feasibility study within six months, followed by a detailed plan and funding proposals within one year, aiming for the initial groundwork to begin by the two-year mark.
Recommendations for the Global North:
Coordinate Joint Emergency Response Mechanisms: AI Safety Institutes in the Global North should set up collaborative emergency response protocols to treat any emergent AI risks with the most vulnerable Global South governments for AI-related crises (e.g., algorithmic failures or AI-driven security threats). These protocols could detail how expertise and resources can be rapidly deployed in a crisis scenario.
Develop Localised Adaptation Blueprints: Require AI Safety Institutes in the Global North to develop “localised adaptation blueprints”26 of their AI safety protocols, which are designed to the context of and in partnership with Global South institutions. These could include modular safety frameworks that account for differing levels of regulatory and infrastructural maturity, risk mapping for regional vulnerabilities, cultural and linguistic customisation of AI systems, and phased implementation strategies aligned with domestic AI strategies.
Form Institutional Partnerships and Fellowships: Develop long-term partnerships between established AISIs in the Global North and new or potential AISIs in the Global South. This could involve knowledge exchange programs, technology transfer agreements, joint research projects, and hosting fellowships for Global South researchers and policymakers.
Analogy: The Global Health Security Agenda (GHSA)27 facilitates knowledge exchange, joint projects, and capacity building to enhance global health preparedness. It could be looked at as a model for partnerships between developed and developing countries.
Objective 2: Coordinate a Global Moratorium on Lethal Autonomous Weapons Systems (LAWS)
Problem: Preventing Tech-Fueled Conflict Hotspots
The development and deployment of Lethal Autonomous Weapon Systems (LAWS) represent an urgent, high-stakes risk, disproportionately affecting the Global South.28 These autonomous systems, which can select and engage targets without human intervention, are poised to reshape warfare in ways that exacerbate existing vulnerabilities. The Global South, often characterised by higher conflict prevalence,29 fragile institutions,30 and the proliferation of non-state actors,31 stands at the frontline of these emerging threats. Without the means to manage or influence the governance of LAWS, these regions could become testing grounds for technologies that are unreliable and unpredictable,32 which destabilise fragile societies and escalate violence.
Call to Action: Facilitate technical workshops and research forums over the next year to generate data and insights that inform global discussions on a LAWS moratorium, with the ultimate goal of building a technical case for future international agreements.
Timeline: Organize the first technical workshop within four months and establish a research consortium to release a comprehensive technical report within one year.
Recommendations for the Global North:
Perform Red Teaming Simulations: Organize red teaming exercises,33 where Global North military and AI safety experts simulate potential scenarios involving LAWS in Global South contexts to better understand the threats and design robust mitigation strategies. The findings should be shared with policymakers to inform regulations.
Delineate AI Safety Institutes’ Role in Risk Mapping: Global North AI Safety Institutes could perform “regional risk assessments” focused on how the spread of autonomous weapon systems could destabilise Global South regions. These risk assessments34 could inform policy recommendations and build geopolitical cases against the spread of LAWS.
Provide Technical Assistance and Information for Weapon Verification Systems: While this still involves technical support, AI Safety Institutes can lead the development of AI-powered verification tools that are designed to detect and monitor LAWS activities in Global South regions. These tools should be deployed with assistance from Global North defence ministries to enhance global monitoring and cover all global conflict hotspots.
Note: Collaborating with defence ministries, these institutes can deploy such tools in regions susceptible to LAWS proliferation, enhancing global monitoring and mitigating potential conflicts. This approach mirrors existing verification mechanisms in arms control, such as the use of remote sensing and open-source data for monitoring compliance with Weapons of Mass Destruction (WMD) treaties.35 For example, these tools could analyse data from various sources to detect unauthorised deployment of Lethal Autonomous Weapon Systems (LAWS), ensuring adherence to established norms.
Objective 3: Leverage AI Responsibly for Achieving the Sustainable Development Goals (SDGs)
Problem: Unlocking AI’s Potential for Global Good
Artificial Intelligence holds immense potential to accelerate progress toward the Sustainable Development Goals (SDGs) in the Global South.36 However, realising this potential requires responsible, well-governed AI deployments that address regional challenges rather than exacerbate them and not merely focus on including underrepresented groups but also make it work for all.37 The Global South faces deeply embedded structural limitations like economic inequality, limited social inclusion, and lack of technical infrastructure.38
Call to Action: Form a working group to draft practical, contextual guidelines for AI projects to achieve SDGs in the Global South, with a timeline for publishing these guidelines and testing them in pilot projects within one year.
Timeline: Establish the working group within three months, with draft guidelines ready for feedback within six months, and begin pilot testing in one year.
Recommendations for the Global North:
Establish Public-Private AI Deployment Guidelines: Draft practical guidelines39 for AI deployments, focusing on minimising negative impacts and maximising SDG benefits. These guidelines should emphasise the use of transparent, explainable AI systems with clear red lines,40 especially in critical sectors like healthcare and agriculture.
Carry Out AI Safety Auditing for Development Projects: Establish auditing mandates for any AI project by development agencies (like USAID41 or the UK’s FCDO) to undergo a pre-deployment safety and ethics audit. AI Safety Institutes should develop these auditing guidelines to minimise the risk of harm in Global South deployments.
Analogy: The Millennium Villages Project, which aimed to alleviate poverty in Sub-Saharan Africa through targeted interventions, encouraged farmers to plant maize (corn) to boost food security. However, this approach faced contextual challenges: farmers encountered difficulties selling surplus maize due to distant markets, leading to post-harvest losses and limited income generation; additionally, maize cultivation required substantial water, which was often inaccessible, especially in regions lacking adequate irrigation infrastructure.42
Develop Ethical AI Assessment Frameworks: Work with Global South governments to develop an “AI for SDGs Ethics Checklist” that ensures AI projects are designed with safety, fairness, and protection of human rights in mind. Provide training for the concepts and tools needed to implement this framework.43 Best practices and case studies could be shared between actors in a multi-stakeholder format.
Objective 4: Safeguard Human Rights, Democracy, and the Rule of Law in AI Governance
Problem: Closing the Regulatory Lag
AI technology is advancing at an unprecedented rate, outpacing the development of regulatory frameworks in the Global South.44 This regulatory lag leaves countries ill-equipped to confront the ethical, social, and economic challenges AI introduces. Populations are increasingly at risk of digital rights violations, unchecked surveillance, and economic exploitation. Principles like openness and explainability are often designed with the Global North in mind, assuming access and agency that may not exist in other contexts. When applied in the Global South, these principles can be impractical or counterproductive, highlighting the urgent need for tailored regulatory approaches that fit local realities.
Call to Action: Set up an exploratory task force to evaluate the implementation of ex-ante Human Rights Impact Assessments for AI technologies, focusing on aligning these mechanisms with local governance readiness levels in the Global South, with findings to be presented within 12 months. Inspiration could be taken from the learnings shared by the team implementing the Chilean Readiness Assessment Methodology.45
Timeline: Form the task force within three months, conduct evaluations over nine months, and publish a report with recommendations by the end of the 12-month period.
Recommendations for the Global North:
Mandate Interoperable Standards for Global Tech Firms: Require tech companies operating in the Global North to publicly disclose how their algorithms are trained, audited, and evaluated, especially when deployed in the Global South. Work towards incorporating Global South concerns and establishing international transparency benchmarks to hold firms accountable. Global North and Global South actors must work together to combine soft and hard-law approaches in a way that establishes meaningful international guidelines while also being cognizant of regional or local regulations.46
Facilitate the Implementation of ex-ante Human Rights Impact Assessments (HRIAs): Encourage Global North countries to adopt ex-ante HRIAs47 for AI systems, similar to impact assessments used for environmental projects, to evaluate the human rights implications of AI deployments. Share best practices with the Global South and promote the integration of these assessments into local governance.48
Work on Pre-Deployment Safety Reports: Mandate that Global North AI Safety Institutes prepare and share comprehensive safety evaluation reports with Global South governments before deploying AI technologies. This ensures that potential risks are flagged and mitigated collaboratively.49 Facilitate early access for Global South policymakers to AI safety evaluation reports by Global North institutes.50 This allows for timely feedback and adaptation, ensuring AI models are better suited to diverse linguistic and cultural contexts.
Objective 5: Mitigate Language and Cultural Bias in AI Systems
Problem: Ensuring AI Speaks Every Language
Current AI models are heavily biased toward the cultural and linguistic norms of the Global North.51 This creates a significant risk and disadvantage for the Global South, where diverse languages and cultural practices are often misrepresented or wholly excluded. Such biases perpetuate digital inequality, systemic racism, and a lack of access to fair AI-driven services. Additionally, the opacity of these AI algorithms makes it difficult to hold developers accountable for the harm caused by biased systems.52 The Global South needs AI that recognises and respects its cultural diversity, yet it currently faces systemic barriers that deepen existing inequalities and marginalise already vulnerable populations.
Call to Action: Commission a series of regulatory stress tests over the next year to identify and address language and cultural biases in AI models, ensuring these findings are incorporated into global AI safety standards.
Timeline: Begin designing stress tests within three months, conduct simulations over the following six months, and present results and recommendations within one year.
Recommendations for the Global North:
Deploy Monitoring and Evaluation Toolkits: Provide open-source toolkits for monitoring AI deployments, including metrics for safety, bias, and performance.53 Offer training programs to empower local regulators and institutions to use these tools effectively.
Perform Regulatory Stress Testing: Encourage Global North regulatory bodies to conduct “stress tests”54 of their AI governance models using simulated Global South scenarios. This would expose potential blind spots and allow for the creation of more resilient frameworks that account for varied socioeconomic realities.
Conduct Cross-Continental Joint Safety Evaluations: Global North AI Safety Institutes should carry out safety evaluations of AI systems designed for projects in the Global South. This collaboration helps both sides: it highlights vulnerabilities and biases that may not be obvious in a Western context (benefiting the Global South) while also refining the safety mechanisms of these AI systems to make them globally robust (benefiting the Global North). For example, a healthcare AI model might perform well in Europe but fail to consider unique disease prevalence or healthcare delivery methods in Africa. Addressing these disparities strengthens the system overall.
Conclusion
The path forward for AI governance must embrace the spirit of working together, for we have far to go—whether it be in building bridges or understanding bottlenecks. The efforts today will define the landscape of tomorrow, making it imperative that AI governance is shaped by comprehensive, culturally sensitive collaborations that reflect our global diversity.
The Global South brings critical insights and unique experiences that can strengthen international AI standards, making them more robust and globally applicable. For the Global North, engaging meaningfully with the Global South is not just an act of fairness but a strategic investment in a safer, more balanced, and resilient future. We must build governance frameworks that are not only ethical and effective but also representative of our diverse world. This requires rigorous risk assessments, culturally attuned rights protections, equitable capacity building, inclusive standards, and preparedness for emergent crises—all woven into collaborations that reflect the unique strengths and challenges of both the Global North and South.
Acknowledgements
We acknowledge the use of the large language model GPT-4 in copyediting this paper.
References
Notably, this report does not go into what Global South countries should do regarding these objectives. This will be addressed in an upcoming report by the authors.
“‘Irrefutable’ Need for Global Regulation of AI: UN Experts | UN News,” September 19, 2024, Link.
UNCTAD, ed., Forging the Path beyond Borders: The Global South (New York/Geneva: United Nations, 2018).
“Seoul Declaration for Safe, Innovative and Inclusive AI by Participants Attending the Leaders’ Session: AI Seoul Summit, 21 May 2024,” GOV.UK.
“Widening Digital Gap between Developed, Developing States Threatening to Exclude World’s Poorest from Next Industrial Revolution,” UN Meetings Coverage and Press Releases.
Jennifer Clapp, “The Toxic Waste Trade with Less-Industrialised Countries: Economic Linkages and Political Alliances,” Third World Quarterly 15, no. 3 (1994): 505–18.
UN Environment, “Basel Convention on the Control of Transboundary Movements of Hazardous Wastes,” UNEP, December 9, 2011.
“Melamine-Contaminated Powdered Infant Formula in China – Update 2,” WHO.
Jane Parry, “China’s Tainted Milk Scandal Spreads around World,” BMJ 337 (October 1, 2008): a1890.
Carolina Gottardo and Nishadh Rego, “The Global Compact for Migration (GCM), International Solidarity and Civil Society Participation: A Stakeholder’s Perspective,” Human Rights Review 22, no. 4 (2021): 425–56.
“UN to Adopt Migrant Pact Hit by Withdrawals,” France 24, December 10, 2018.
Omwansa, T. “M-Pesa: Progress and Prospects,” Innovations / Mobile World Congress (2009): 107–123.
“The Long-Run Poverty and Gender Impacts of Mobile Money,” Science.
“PNAI Report,” Internet Governance Forum.
Ibid.
Wragg, David W. A Dictionary of Aviation, 1st ed. (Osprey, 1973), 164.
“Pages – Safety Management,” ICAO.
“[2410.18417] Large Language Models Reflect the Ideology of Their Creators,” arXiv.
“Unmasking the Bias in Facial Recognition Algorithms,” MIT Sloan.
“AI Safety Summit: Confirmed Attendees (Governments and Organisations),” GOV.UK.
“AI Seoul Summit: Participants List (Governments and Organisations),” GOV.UK.
Kristina Fort, “The Role of AI Safety Institutes in Contributing to International Standards for Frontier AI Safety” (arXiv, September 17, 2024).
“2024 Government AI Readiness Index” (2024).
“Understanding the First Wave of AI Safety Institutes: Characteristics, Functions, and Challenges,” Institute for AI Policy and Strategy.
“AI Safety Institutes: Can Countries Meet the Challenge?” OECD.AI.
Localized Adaptation Blueprints are modular frameworks that customize AI safety protocols for the Global South, addressing region-specific risks like data misuse or biased algorithms.
“The Global Health Security Agenda (GHSA): 2020–2024” (n.d.).
“The Global South and Autonomous Weapons Controls,” Arms Control Association.
“ACLED Conflict Index,” ACLED.
John Idriss Lahai and Helen Ware, eds., Governance and Societal Adaptation in Fragile States (Springer, 2020).
International Law Editorial, “Exploring the Impact of Non-State Actors on Global Governance,” World Jurisprudence, December 1, 2024.
Whitepaper by Alycia Colijn and Heramb Podar, “Technical Risks of (Lethal) Autonomous Weapons Systems” (n.d.).
Tessa Baker, “What Does AI Red-Teaming Actually Mean?” Center for Security and Emerging Technology, October 24, 2023.
“ISO 31000 — Risk Management,” ISO, December 10, 2021.
Veronica Borrett et al., “Science and Technology for WMD Compliance Monitoring and Investigations,” November 12, 2020.
Brigitte Hoyer Gosselink et al., “AI in Action: Accelerating Progress Towards the Sustainable Development Goals” (n.d.).
Alan Chan et al., “The Limits of Global Inclusion in AI Development” (arXiv, February 2, 2021).
“The ‘AI Divide’ between the Global North and Global South,” World Economic Forum, January 16, 2023.
Guidelines should prioritize data sovereignty, ensuring local control and ethical data use while fostering regional capacity-building to reduce reliance on external actors.
“The AI Red Line Challenge,” Tech Policy Press.
USAID acknowledges the potential risks associated with deploying AI in developing-country contexts in its “Artificial Intelligence in Global Development” report; as of May 2023, only two AI systems were in use within its foreign assistance programs according to an inventory submitted to the OMB.
“The Idealist by Nina Munk,” Penguin Random House.
D. Leslie, C. Rincón, M. Briggs, A. Perini, S. Jayadeva, A. Borda, S. J. Bennett, C. Burr, M. Aitken, M. Katell, J. Fischer Wong, and I. Kherroubi Garcia, AI Sustainability in Practice, Part One: Foundations for Sustainable AI Projects (The Alan Turing Institute, 2023).
Collingridge, David. The Social Control of Technology (St. Martin’s Press / Pinter, 1980).
“7 Lessons from Implementing the RAM in Chile,” UNESCO.
“PNAI Report,” Internet Governance Forum.
Hickok, Merve, Marc Rotenberg, Christabel Randolph, and Sneha Revanur. “Artificial Intelligence and Human Rights.” Statement to the U.S. Senate Judiciary Committee, Subcommittee on Human Rights and the Law, June 13, 2023.
Yoshua Bengio, “International Scientific Report on the Safety of Advanced AI – Interim Report” (n.d.).
Markus Anderljung et al., “Frontier AI Regulation: Managing Emerging Risks to Public Safety” (arXiv, November 7, 2023).
“AI Safety Institute Releases New AI Safety Evaluations Platform,” GOV.UK.
“[2410.18417] Large Language Models Reflect the Ideology of Their Creators,” arXiv.
“The ‘Missed Opportunity’ with AI’s Linguistic Diversity Gap,” World Economic Forum.
Rachel K. E. Bellamy et al., “AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias” (arXiv, October 3, 2018).
Stress tests could involve sandboxes with different environments, scarce resources, or simulations of how SDG metrics perform.
In September and October of 2025, the European Commission called on European citizens and organizations for evidence on their Digital Omnibus initiative, an attempt to simplify rules around digital innovation in order to foster innovation. Below, you can find our response to the call. On November 19, the Commission subsequently published their plans for the Digital Omnibus.
Simplification should however not lead to weakening of regulatory safeguards that are in place. The AI Act is complex structurally and linguistically, thus an attempt at simplifying for efficiency and effectiveness is welcome but needs careful implementation strategies so as not to lose precision and specificity and cause ambiguity, or less stringent protection of human rights. We suggest careful legal drafting and strong inter-institutional cooperation to avoid losing the regulatory basis of such Acts due to simplification.
In striving for simplification, Responsible AI should not be governed by a culture of tick-the-box ethics. That brings us to a few remarks.
Firstly, most entrepreneurs struggle with the complexity of rules, as well as the question of which rules apply. Therefore, we believe a solid communication strategy (possibly in collaboration with national compliance authorities) is a vital part of simplification. These could include visual product-launch journeys that include details on both the AI Act and the subsequent guidelines on prohibited and regulated systems that have been introduced.
Secondly, we encourage a sector-specific digital omnibus and suggest that the recently announced AI Act Advisory Forum sets up a working group around simplification, including representatives of advocates for entrepreneurial affairs (e.g. Chambers of Commerce and advocacy organizations).
Thirdly, we recommend specific guidelines for software developers and a question list that entrepreneurs can use when working with software partners. Most SMEs will not develop advanced AI models but rely on software embedding AI or build on foundational models. While the General Purpose AI Code of Practice guides the foundational models, tailored guidance for developers would facilitate AI Act adoption.
Fourthly, we believe the striving for simplification should not just be an ambition of administrative simplification, but an attempt to help small-cap entrepreneurs with the ethical implementation of AI in their business models, as most would not have the resources to conduct holistic risk analyses. Therefore, we argue for standardized, easy-to-access-and-use ethical tools, such as: open-access AI-ethics impact templates, shared databases with best practices, and common audit frameworks within sectors. These resources could be shared via the AI on Demand resource database initiative; however, AIoD would need rigorous simplification to make it user-friendly as well.
Fifthly, for the upcoming Digital Fitness Check of Consumer Laws, we would welcome recommendations in the field of contextual advertising, which reduces pressure on data centers and computing power, improves privacy, and — research shows — leads to higher conversion for commercial parties. It also mitigates monopolization risks in the field of online tracking and consumer data. For the Digital Fitness Check that will analyze the cumulative effects of the simplification, we encourage the Commission to not only analyse the reduced administrative burden, but also include ethical performance indicators, e.g. transparency of AI-related decision-making and notifications of bias and harm caused by AI systems.
Lastly, on data legislation, we encourage the Commission to prioritize, or where possible require, federated learning and on-device processing. These protect privacy, benefit entrepreneurs, and support sustainability by reducing computing power demand, aligning naturally with GDPR objectives. As the AI Act requires non-discrimination, accuracy, and traceability in datasets, we advocate for more explicit provisions ensuring datasets meet these standards. This would accelerate ongoing standardization across sectors and member states.
We deeply appreciate the Commission’s ongoing work to ensure responsible AI development and stands ready to support the implementation.
The Next GenAI Forum: Preparing for Tomorrow’s AI Governance Today convened youth leaders, policymakers, researchers, and civil society actors on January 30, 2026. The aim was to examine how artificial intelligence is reshaping safety, labor, and opportunity with a deliberate focus on elevating Global Majority and youth perspectives in governance conversations. The event was hosted through participation hubs in Accra, Ankara, Astana, Montreal, and online across our international network. The hybrid format allowed cross-regional dialogue grounded in local realities, with ~230 attendees joining from 15+ countries. The forum centered on safe & trusted AI, inclusion and social empowerment, and youth futures in an AI-driven economy, using panels and a closing workshop to surface agency gaps and pathways for open engagement. This post-event brief synthesizes key youth-driven insights and recommendations intended to inform the India AI Impact Summit (2026) and strengthen ongoing policy contributions on the global stage.
1) Fortify Local AI Governance Capacity and Public-Interest Infrastructure
AI systems are only as safe and fair as the communities they serve, yet their rapid scaling is outpacing local institutions’ capacity to respond, widening the gap between deployment and effective governance. Without sustained investment, AI frameworks risk becoming symbolic, i.e., adopted in principle but unenforceable in practice, which highlights the need for local technical expertise, public-interest infrastructure, and tools to interpret and apply governance in context. Key barriers include limited readiness to assess AI in public services, difficulty in adapting global frameworks to local contexts, and the underrepresentation of youth in decision-making spaces.
Strengthening local capacity requires:
Invest in local technical expertise, open data access, and audit mechanisms to ensure accountable AI deployment.
Embed youth voices and cross-sector partnerships in governance to align with community needs.
Build AI literacy and workforce foresight to prepare communities for labour market shifts and emerging opportunities.
2) Implement Risk Scanning and Response Through Capacity Building
Warning signs are often visible first to researchers, students, and practitioners close to deployment. Yet these signals rarely travel into decision pipelines and, without institutional homes, dissipate.
To operationalize this link, participants, we should:
Establish standing horizon-scanning units within ministries to produce recurring assessments, maintain watchlists of emerging systems, and initiate regulatory reconsideration when predefined indicators are met.
Enable structured cross-border exchange via shared reporting channels and regional coordination that can surface patterns before they escalate.
Establish clear requirements for independent impact assessments before deploying AI systems and throughout the AI lifecycle, in both the private and public sectors.
3) Build Agile Regulatory Approaches to Reinforce Societal Resilience in the Face of AI Risks
If a system can change every month but a rule changes every five years, real authority migrates away from regulators. Moreover, the long-term effects of AI on society (such as its impact on democratic institutions) remain to be seen and fully understood. Without regulatory agility and societal adaptation, governance frameworks risk becoming symbolic and disconnected from how AI actually affects people on the ground. Even where capability appears limited, scale alone can amplify harm.
To reinforce societal resilience, we should:
Design flexibility into legal instruments, such as sunset clauses and automated compliance, that can prevent rules from lagging behind capability shifts.
Mandate AI systems used in public or high-risk private sectors provide clear documentation and meaningful contestability mechanisms, including performance limitations.
Install safeguards in the digital information environment, such as watermarking and provenance measures, to protect the information commons.
AI’s trajectory remains uncertain. Technical progress, labor impacts, and governance needs may evolve in ways that are difficult to anticipate from today’s vantage point. However, when prediction is limited, adaptability becomes the core asset, and broadening who contributes to governance strengthens the system’s ability to respond. The next phase of global AI governance must be shaped in coordination with people entering the field today.
Organizing Committee
Heramb Podar (Lead Writer), Meriem Mehri (Lead Writer), Adrian Klaits, Joseph Awuah
