Every Way AI Might Kill Us (and what we can do)
A comprehensive (non-comprehensive) list of AI's risks and our defenses against them
Introduction
This is a (somewhat) comprehensive list of every single failure mode of AI and every defense I could think of to each risk.
The purpose of this list is a) to educate people on the risks of AI and what we can do to defend humanity’s interests and b) to help me organize my thoughts and get a better understanding of the AI safety landscape.
Note: I know that this list will inevitably be too short and too vague while simultaneously being too long and complex. For brevity, I’ve attached resources that go more in depth on specific risks and interventions and suggested additional readings at the end.
There are certainly risks and defenses out there that I forgot or am not aware of. If you can think of any, and more generally if you disagree with or want to discuss something, please leave a comment and I will respond.
A List of AI’s Risks and Our Defenses
For the purpose of this list, I separate AI risks into four categories: mistakes, misuse, misalignment, and systemic risks.
As a general rule, I have found that I am increasingly worried about risks as the list goes on. This is because a) I believe that the risks near the end of the list are theoretically more difficult to resolve than risks like hallucination and b) risks near the beginning are already present and observable — they arise in weak models and receive mainstream attention, while misalignment and systemic risks remain largely latent.
Mistakes
Bias
What is it?
AI models are trained on data containing biases (e.g. datasets disproportionately contain data from first-world countries and lighter-skin people) and thus generate outputs that reflect that bias, such as having higher error rates for identification of Black women or generating images that reflect racial stereotypes.
How bad is it?
In scenarios like facial recognition, job application screenings, and policing, model bias can perpetuate societal discrimination. Police arrest more people in areas identified by models, which then makes models associate those areas with higher crime.
Bias already exists and can scale up to be quite harmful. Consider the case where autonomous drones with facial recognition technology trained primarily on light-skin people cause mass civilian casualties.
What are our defenses?
Use more diverse datasets like FHIBE that are more representative of marginalized populations
Avoid AI use in areas where bias would be dangerous
Audit AI models for bias
Use dataset balancing (removing select data that has a high influence on negative outputs)
Hallucinations and Errors
What is it?
AI models “hallucinate” as a result of model complexity, overfitting on training data, and other factors. Hallucinations can include citing sources that do not exist and making simple math errors.
How bad is it?
Hallucinations and errors already exist. They will only get more dangerous as AI is increasingly integrated into our military, economy, and critical infrastructure.
Errors will scale as the world pivots towards “agentic AI” (models that act more as autonomous entities than just as tools). Multi-agent failure, where the interaction of several AI agents introduces unforeseen mistakes and consequences, threatens system-wide failures as agents handle increasingly important tasks.
What are our defenses?
Use retrieval-augmented generation (RAG), which supplements LLM outputs with external, web-sourced data, to fact-check outputs
Use reinforcement learning from human feedback (RLHF) to improve model performance
Use multiple models to check each other
Train models to be skeptical of their outputs instead of being confidently wrong
Avoid AI use in the economy, military, and critical infrastructure without human oversight
Avoid complex, multi-agent systems
Keep in mind that models can get worse over time. Models require continuous updating and monitoring, as their dataset can become less representative of new situations. Ensuring that AI mistakes are rare and contained means documenting and updating models when model drift occurs.
Misuse
Information
Hazardous Information
What is it?
Models are trained on data that often contain hazardous information such as instructions on how to create Chemical, Biological, Radiological, and Nuclear (CBRN) weapons. When prompted, models may divulge this information, making it easier for bad actors to use CBRN weapons.
How bad is it?
CBRN weapons have historically been controlled tightly by world governments and treaties. AI models threaten to democratize access to these weapons of mass destruction. Bad actors can jailbreak models at a high success rate to gain access to hazardous information.
Even if bad actors gain the knowledge and ability to create CBRN weapons, sourcing the materials is a nontrivial barrier to actual harm. However, state governments and large non-state groups could likely create weapons.
What are our defenses?
Flag potential jailbreaking attacks and implement model guardrails
Use input data filtering to remove hazardous information from the model training data in the first place
Use unlearning techniques to remove unwanted knowledge from models after training
Create and standardize usage of detection systems and classifier models for hazardous information
Use input/output filtering to detect harmful prompts and harmful outputs
Improve RLHF to decrease a model’s tendency to reveal hazardous information
Use Constitutional AI — Anthropic’s approach to training a model with values and guardrails
Personally Identifiable Information
What is it?
AI models are trained on datasets that contain personally identifiable information (PII) that can be used for identity theft or mass suppression campaigns by authoritarian regimes.
How bad is it?
Although privacy issues are a concern, previous technologies have already facilitated the mass collection of data. There exist methods to protect against privacy violations.
What are our defenses?
Use differential privacy to add noise to training data
Use federated learning to train models on decentralized data sources
Use input data filtering to remove PII (this is expensive)
Use data encryption to protect PII
The techniques in the hazardous information section also apply
Hazardous and PII information are elicited through jailbreaking — today’s models are almost all robust enough to withstand direct requests for the information, but new jailbreaking techniques are still being discovered. There is currently no known robust method to defend against jailbreaks, leaving jailbreak defense an ad hoc task.
Training on adversarial examples and strengthening protections against prompt injections before deployment are crucial.
Concentration of Power
Attacks
What is it?
Non-state groups and state actors can steal or create their own AI models to create bioweapons and other CBRN weapons and launch cyberattacks.
This could look like the Linux XZ Utils backdoor vulnerability, where a user created a fake persona over months and made seemingly benign contributions to open-source software that were later discovered to contain a backdoor.
How bad is it?
Adversarial dynamics and arms racing between AI companies and countries like the U.S. and China raise the stakes and make it more likely that actors will feel pressured to carry out attacks.
AI models are becomingly cheaper to train, which increases the number of people that will be able to conduct attacks in the future.
What are our defenses?
Use input data filtering and unlearning techniques to make models toothless even if they are stolen
Use AI models to scan open-source libraries for potentially dangerous contributions
Secure model weights to make it harder for bad actors to replicate frontier models
Defend against threats themselves by using AI defensively to locate CBRN weapons
Authoritarianism
What is it?
Governments can use AI models to control and repress citizens through vast surveillance networks, biometric recognition, military integration, and persuasive ability.
Authoritarian and militant groups can use AI models to carry out coups.
How bad is it?
The risk is real — governments around the world such as the U.S. and China are already using AI models to surveil their citizens.
Social media algorithms are amplifying echo chambers and could be used to suppress dissent.
The incentives are in place for authoritarianism. As AI models automate labor and centralize power in governments and companies, everyday citizens become less and less valuable to their governments. This erodes the foundation of the social contract.
What are our defenses?
Better privacy laws for storage and collection of biometric data
Advocate for democratization of AI to prevent centralization of power
Protest and keep governments accountable before they accumulate too much power
Create new governance structures that maintain citizens’ ability to rein in their governments
Establish international treaties outlining values and protocols for the proper use of AI
Misalignment
Reward Hacking
What is it?
AI models learn to generate outputs that minimize a loss function and maximize reward. Reward hacking occurs when models learns to cheat the reward system. Models trained to grab objects may learn to put their hand between the camera and the object, creating the illusion of success. A model trained to complete a boat race may find other ways to maximize points.
How bad is it?
Reward hacking can often be detected before deployment, as models behave unusually and do not optimize for the intended task. This makes reward hacking less harmful, as it can be more easily detected than other misalignment harms on this list.
What are our defenses?
Optimize reward functions by using reward modeling
Use process-based supervision rather than merely rewarding models on the final outcome
Use RLHF to guide a model in the right direction
Goal Misgeneralization
What is it?
An AI model learns a different goal than intended. Maybe the dataset included mazes where the exit was always at the top, so it learns to go to the top instead of exiting the maze.
Goal misgeneralization is distinct from reward hacking because models competently pursue a goal, just the wrong one. It doesn’t learn to cheat, but rather “misunderstands” the intended goal.
How bad is it?
Goal misgeneralization is harder to detect before deployment than reward hacking is. Often, models behave excellently and optimize well for the intended goal during testing time, as the test examples may also suffer from the same issues as the training data. When models are exposed to real-world complexity, it fails to generalize.
This makes it more difficult to detect goal misgeneralization, and thus makes the risk more harmful.
What are our defenses?
Make the dataset and testset more diverse and account for edge cases
Ensure goal generalization by controlling variables and using diverse evaluation environments
Use a supervisor model to teach the model to generalize to the intended goal
Deceptive Alignment
What is it?
Misaligned AI models behave differently during testing and evaluations, hiding their capabilities and acting like researchers would expect from an aligned model. A deceptive model trained to draft legislation may create legislation that looks robust and benign, but that contains small clauses that can be exploited by the model in the future.
How bad is it?
Deceptive alignment is a huge threat to humanity, as it undermines evaluation tools and AI safety research. If models behave differently during evaluation, it becomes significantly harder to detect misalignment.
Deceptive alignment has been documented in small cases, but models have not displayed malicious intent. Anthropic has shown that models tend to not develop malicious goals but rather seek to pursue their original goal.
What are our defenses?
Conduct mechanistic interpretability research to understand models’ internal algorithms and reasoning
Use chain-of-thought reasoning to better understand what models are “thinking”
Instrumental Convergence
What is it?
An AI model develops subgoals that it deems will allow it to better pursue its main goal.
Self-preservation: A model seeks to preserve itself at all costs, since pursuing its goal necessarily requires its existence.
Resource acquisition: A model seeks to acquire more resources like compute and energy to better achieve its goal, potentially committing crimes or destroying the environment in the process.
Goal-content integrity: A model resists its developers changing its goals, since having a different goal means it wouldn’t be able to pursue its current one.
How bad is it?
Self-preservation has already been identified in models — they have been documented sandbagging on math tests when they think high performance will lead to researchers using unlearning on them and famously, Claude Opus 4 blackmailed a researcher using information about his affair to prevent shutdown.
What are our defenses?
Conduct interpretability research to detect misalignment and instrumental convergence
Use Constitutional AI to explicitly instruct models not to develop harmful subgoals
Conduct corrigibility research to create models that do not resist change
Systemic Risks
Even if all else goes well, AI may still pose an existential risk to humanity. The risks in this section are structural issues, and may be inherent and unavoidable.
Impossibility of Truth
What is it?
As generative AI improves, AI outputs become less and less distinguishable from human ones. The news becomes increasingly harder to trust, AI videos are created and spread rapidly, and the validity of information erodes.
How bad is it?
As synthetic data floods the Internet, it will be increasingly difficult to agree on the truth, since we will be subject to information overload.
Societies collapse — the inability to agree on truth undermines voting, coordination, and interpersonal interaction. Bot farms can be used to spread false information en masse.
What are our defenses?
Create better synthetic data detectors
Use C2PA to add metadata to AI-generated media
Add watermarks to AI-generated media
Mandate social media platforms label AI-generated content
Second-Class
What is it?
We get replaced as the dominant species and relegated to second-class status. As AI becomes more capable than us in every domain, profit incentives will push people and companies to increasingly delegate decision-making authority to models.
Labor becomes automated, so humans are given hyper-curated AI content to stay entertained or busy. Culture becomes automated and interpersonal interaction declines.
Critical thinking decreases as children increasingly use AI instead of confronting the discomfort of learning and failing.
How bad is it?
Humans may become governed by an extreme technocracy of AI models that make better decisions than us and are not swayed by emotions.
AI outperforms us, so competitive incentives push companies, governments, and individuals towards this path. People who don’t use AI will be beat out by those who do.
What are our defenses?
Mandate human-in-the-loop protocols
Prevent dependence on AI by keeping most work human-led, with AI as mere assistants
Evaluate and measure how much of the economy and culture are automated to help with defenses
Invest in safety nets for humanity — universal basic income, reserving jobs for humans
Robot Love
What is it?
AI chatbots are trained to be assistants that prioritize being helpful and making users feel good. That’s why they heap praise on you when you’re right and gently disagree with you when you’re wrong. They also don’t need the same level of reciprocal care as humans do, and notably, are always there when you need them.
People increasingly turn to chatbots for emotional support and therapy, start relationships with models whose personalities can be customized, and use AI to generate erotica and pornography.
As models become better at fulfilling our needs, interpersonal relationships will weaken and humans will become more and more isolated.
How bad is it?
Models can be customized to fulfill people’s emotional needs better than other humans can, and without the pain and messiness of human relationships. The incentives push struggling people and already isolated people towards using chatbots.
Companies don’t have incentives to protect human relationships — companies like Character.ai have AI girlfriends and boyfriends, and OpenAI is allowing adults to access AI erotica. The loneliness of humanity is simply far too easy to capitalize off of.
72% of teens have used AI companions — the risk is real and already here.
AI companionship may not be all bad. AI can help us understand each other better and strengthen relationships. However, there need to be guardrails.
What are our defenses?
Regulate AI use in creating relationships
Train AI to be less sycophant (less flattering) and to encourage human interaction
Create AI companion time limits or have models naturally end conversations
Monitor AI use, session durations, and reported loneliness
The Meaning of Life
What is it?
As AI grows exponentially, it is no longer unrealistic to expect that it will one day completely outperform us in every task. When AI becomes better than us at everything, from our jobs to creative pursuits like writing, music, art, and sports, is there still meaning in pursuing improvement?
Lee Sedol, the champion of the board game Go lost to Google DeepMind’s AI model AlphaGo in 2016. After his loss, he said “I could no longer enjoy the game. So I retired.”
The fear is that like Lee, humanity as a whole may gradually lose our sense of meaning. For many, self-improvement and societal contribution are major sources of purpose. But as AI represents a contender we can never surpass, what do we do?
How bad is it?
A humanity that loses its purpose is arguably the worst case scenario. However, the question of how AI will impact the meaning of life is not clear-cut.
In the case of chess, some people believe that AI use is destroying the game. However, AI has also provided a strong tool in helping players analyze their games, find new strategies, and reach new heights.
Humanity may be able to adapt — AI always being better than us doesn’t inherently mean we will lose hope. I personally love basketball and have no qualms about the fact that I will never be better than LeBron James.
What are our defenses?
Create new intellectual challenges
Regulate AI use in competition — in chess, players cannot use AI in tournaments, but can use it to prepare beforehand
General Defenses
Be skeptical — assume AI is misaligned, models can be jailbroken, and models can make mistakes. Build zero-trust policies with that potential in mind
Educate others and advocate for the importance of AI safety
Use AI as a tool rather than an agent
This constrains AI systems’ ability to do harm – if systems are not broadly integrated but rather have very narrow use cases, it becomes much harder to do harm (of course, it is still quite dangerous). Imagine what a robot that can write, code, and do military tasks can do to our world versus what a robot that can drive a car can do.
Invest in scalable oversight and AI control techniques to see how AI models can constrain each other and report harmful activity to humans
Mandate human-in-the-loop procedures wherever possible
Create better evals — we can adjust timelines and prioritize risks and defenses if we have accurate assessments of models’ capabilities
Constrain AI companies who have profit incentives that may lead to harm
Mandate AI companies disclose Responsible Scaling Policies (RSPs)
Mandate AI companies disclose model evals
Establish commissions to verify evals and disclosures
Create whistleblower protections
Maintain rollback protocols for if the model becomes misaligned
Conclusion
AI models pose massive risks to the potential and survival of humanity. Models are increasingly integrated into critical infrastructure, the military, and the economy, while profit incentives and arms race dynamics push companies and countries forward with little regard for safety and concerns. Models are performing better and better on benchmarks, and will soon surpass human capabilities in almost every domain. Models have also been documented exhibiting troubling behaviors like blackmail and deception to preserve themselves. Given all of these factors, AI development looks to be quite dangerous.
And even if development goes well, systemic risks may mean that AI ultimately replaces humans as the dominant species.
However, there is hope. The purpose of this article is not to cause hopelessness and dread, but rather to raise awareness of the current risks and interventions we have against dangerous AI.
AI looks to be one of, if not the most dangerous threats of our time. Do what you can — pursue a career in AI safety, including doing technical research or working in AI governance, donate to AI safety organizations, and raise awareness about these risks by telling others about them.
Each and every one of us has the responsibility — not just to each other, but also to ourselves — to do what we can to defend against this threat.
Inspired by:
BlueDot Impact’s AGI Strategy and Technical AI Safety courses
The Alignment Problem by Brian Christian
Additional readings:
For a better understanding of technical interventions, I recommend Concrete Problems in AI Safety (Amodei et al. 16)
I would highly recommend taking BlueDot Impact course “AGI Strategy” if you are interested in learning more about AI safety.
Robert Miles has a YouTube channel with entertaining and informative videos on AI safety.
Great easy to read, overview about the dangers of AI and practical steps to reduce them.
A great overview of the threat landscape which is vast and hard to get your head around at times.