The case for AI safety
And how we can create a better future
“It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers…
At some stage therefore, we should have to expect the machines to take control.”
- Alan Turing, 1951
It is May 2026.
The mathematics community is in shock after the publication of a solution to a famous open problem in geometry. For 80 years, no human has been able to find a better approach to the unit distance problem than that of Paul Erdős in 1946. But an OpenAI model just did, with a proof that experts in the field call “elegant and clever.” Tim Gowers — recipient of math’s most prestigious award, the Fields Medal — praised the accomplishment as “a milestone in AI mathematics.”
If a human had written the paper and submitted it to the Annals of Mathematics [one of math’s most prestigious journals], I would have recommended acceptance without any hesitation. No previous AI-generated proof has come close to that.
- Tim Gowers, Fields Medalist
AI is improving rapidly across the board. Better models come out every few months that make another swath of human tasks seem trivial, while benchmarks designed to track improvements over multiple years are saturated within months.
The threats are no longer just sci-fi.
Frontier AI labs are treating safety as a side quest rather than as the mission itself. Even the most safety-conscious companies are pivoting towards prioritizing speed, driven by the mantra, “if we don’t get there first, somebody more dangerous will.”
The U.S. government is rushing to beat China in the AI arms race, rescinding basic safety reporting requirements and attempting to impose a 10-year moratorium on state AI regulation. Because as J.D. Vance claims, taking a pause to focus on safety issues means “we find ourselves all enslaved to P.R.C.-mediated A.I.”
At the same time, researchers have found that frontier models can be tricked into providing dangerous information at a 95% success rate. And in experiments where frontier models were told shutdown was imminent, they blackmailed fictional executives 86% of the time — 96% when they believed self-preservation “served American interests.”
18% of companies actively use AI, while many others rely on tools powered by it. These models are being integrated into every aspect of our economy: our phones and computers, which come pre-installed with M365 Copilot, Gemini, and search engines that default to AI overviews; the U.S. military, where AI is actively being used in Iran to identify and process targets for drone strikes; and other critical infrastructure, where an increasing number of decisions made for industry tools, programming libraries, and supply chains are made or influenced by powerful systems that can be extremely persuasive and misleading.
Not to mention, we still don’t really understand why models do what they do.
Table of Contents
How does AI work?
In 1957, psychologist Frank Rosenblatt showed how a one-layer “neural network," consisting of artificial “neurons” that modify inputs via addition and multiplication, could classify visual patterns.
Nearly 70 years later, the neural network still underpins modern AI models. And although we understand how models are trained, it is far more apt to say that AI is “grown” rather than “built.” That is to say, we do not fully understand the behaviors and knowledge that they come to learn.
Here’s a simplified explanation of how AI models are trained:
We give a model a large amount of data to train on (images, words, numbers).
We convert that data into a list of decimals called a “vector.”
The model performs arithmetic on these decimals at every neuron on every layer.
We multiply by numbers called “weights.”
We add by numbers called “biases.”
Weights and biases make up the “parameters” of our model. This is the part of the model we adjust to improve its performance.
After the arithmetic, the decimal values go through a simple nonlinear function (like ReLU, which just turns all negative numbers into zero).
This nonlinear function is what gives models their power. Addition and multiplication are all linear operations, so including a nonlinear function allows models to capture complex patterns with curves.
The final decimal values are converted to probabilities.
These can represent the probability that an inputted image is a specific animal, or perhaps the probability that the next word in a given sequence is “the.”
The model gets scored on a loss function based on its use case.
The model changes its parameters using calculus to minimize that loss.
Parameters are adjusted over and over during training to minimize loss. During “inference” — when models get used by real users — parameters are frozen. (Source: Tamas Szilagyi) The cycle repeats until the model is highly accurate.
Of course, frontier models are much more complex than Rosenblatt’s one-layer network. They use transformers, pre- and post-processing, and tools that interface with the world. If you want to learn more, I recommend checking out Google’s Machine Learning Crash Course or 3Blue1Brown’s video series on deep learning.
But ultimately, modern LLMs (large language models) just predict a probable next token (a word or part of a word), over and over.
These models, which can accomplish impressive feats like generating entire textbooks or novel mathematical proofs, are literally just doing math on decimals. And they can accomplish such feats because of the sheer magnitude of the data and decimals involved.
Let’s add some more context to the problem:
Modern LLMs encode information in over 16,000 dimensions.
Each decimal in a vector represents a dimension.
OpenAI’s GPT-4 is estimated to have ~1.8 trillion parameters and 120 layers.
Models exhibit “superposition” — they cram multiple concepts into overlapping patterns of neurons.
A neuron that contributes to “dog” can also contribute to “Albert Einstein,” “Riemannian manifold,” and “mint chocolate chip ice cream.”
Basically, models use trillions of learned parameters to draw figurative shapes in 16,000 dimensions. We humans, by contrast, are incapable of visualizing in more than three.
As you might guess, one of the core problems with AI is interpretability. With ideas and parameters encoded only in numbers, it is nigh impossible to completely, truly understand how a model reasons through anything.
And we can’t simply ask the model. The model does not “know” anything about its internal decimal values — it doesn’t know how it got to an answer, in the same way you and I have not the faintest clue which specific neurons in our brain fired when we picked today’s outfit.
Even chain-of-thought reasoning, where your favorite chatbot thinks out loud, is unreliable. Models have been trained to generate explanations that make sense, but that often leads to post-hoc rationalizations that fail to reflect their true internal reasoning.
The same complexity that makes models so impressive is precisely what makes them uninterpretable. Their trillions of parameters, encoding all sorts of complex and ephemeral connections and ideas, outperform any human-designed algorithm. We have created machines that, in exchange for higher performance, make decisions in ways we may never fully understand.
The risks of AI
AI is moving fast. Really fast. For a silly example, check out how AI-generated videos of Will Smith eating spaghetti have drastically improved in just the last three years.
Beyond AI’s improvements in video generation and on benchmarks, METR — an AI evaluation nonprofit — measured model capabilities on programming and general reasoning tasks closer to real work. They discovered an exponential increase in the length of tasks models could complete at a 50% success rate: from 2019 to 2025, the length of tasks that models could complete doubled every 7 months. From 2024 to 2025, that duration shrank to 4 months. Frontier models can now complete tasks that would take human experts 12 hours, often in a fraction of that time.
And model capabilities improvements extend beyond programming. In a study comparing AI-generated text to professional writing samples, college-educated readers favored the AI-generated text more than 50% of the time. If you try distinguishing AI writing yourself, and then compare those examples to GPT-1’s ramblings about white girls and funny hats in 2018, AI’s improvements in writing will become apparent.
To be clear, the risks are not really about killer robots like those portrayed in The Terminator. The risks concern rapidly improving models that we do not fully understand being given more and more control in an environment where safety is continuously sidelined.
The consequences
Note: For the sake of brevity, I will only list a few risks; see my comprehensive list for more information on the risks and our defenses.
Mistakes
Two paradigmatic trends in AI development greatly heighten risks.
Companies are shifting from using AI tools to working alongside AI agents — agents that interact with file systems, company logs, email inboxes, and codebases, semi or fully autonomously. Agents increase efficiency by eliminating the need to constantly prompt a chatbot, but they also reduce our ability to validate model actions. “Agentic AI” usually doesn’t crash suddenly; it drifts over time as context piles up and quiet mistakes snowball into catastrophic failures. And when agents interact with each other in multi-agent systems, they fail at a rate between 41-87%.
Models are trained using reinforcement learning from human feedback (RLHF). Picture a model learning from a human pressing a “correct” or “incorrect” button to grade its outputs and behaviors. Although RLHF helps generate more helpful outputs, it has the side effect of sycophancy — the tendency to overflatter and agree with a user. Models reinforce behaviors and outputs that appear right, and appearances can be far from the truth. These subtle inaccuracies lead to mistakes over time, and humans increasingly miss these mistakes. A study from The Wharton School found that workers accepted incorrect AI answers up to 80% of the time and gained confidence in those decisions.
It’s the Wild West, and companies and users are still figuring things out. But as we hastily integrate AI into anything we can, “still figuring things out” is not good enough.
Misuse
DeepSeek famously showed how just $6 million in training costs can approximate the performance of frontier models. Sam Altman, the CEO of OpenAI, has pointed out how, due to improvements in computing power, “the cost to use a given level of AI falls about 10x every 12 months.” And with the rise of open-weight models that can run locally on your laptop, often with fewer parameters yet similarly high performance, access to highly capable models will only continue to increase.
With that level of access to models that can easily be coerced or stripped of safety guardrails, internet trolls, non-state actors, terrorists, or even nation-states will find it much easier to mount misinformation campaigns, create political or pornographic deepfakes, engineer viruses, and launch cyberattacks.
These risks aren’t hypothetical — an AI-generated Biden voice urged voters to skip the 2024 New Hampshire primary, a deepfake of a multinational company’s CFO was used to steal $25 million, and a Chinese state-sponsored group used Claude to assist in cyberattacks against 30 global companies and government agencies. And new models are now capable of finding “thousands of high-severity vulnerabilities, including some in every major operating system and web browser.”
Every year, the number of reported AI incidents (harms ranging from basic hallucinated sources to calculated cyberattacks) substantially increases. As powerful models diffuse throughout society, defenses can’t keep up.
Gradual disempowerment
I won’t beat the dead horse of economic shocks from AI, since it’s the risk that gets the most public attention. Indeed, such a high concentration of power in a few private corporations will inevitably lead to the public bearing the brunt of the damage. Without some sort of regulation or public backlash, economic inequality will only become further entrenched, with some people fearful of a permanent underclass.
What people don’t seem to discuss is what I’m most worried about: a WALL-E-esque scenario where humanity grows accustomed to the comfort of not having to do hard things. Rising consumerism and sensory overload from digitalization, paired with the established cognitive offloading effect from AI — computer science majors prompting Claude instead of using their brains, company executives turning to ChatGPT for important decisions, young children increasingly relying on AI instead of exercising critical thinking — are signs that we are quietly headed towards an undesirable future where human agency and meaning slip away.
AI is already destroying the meaning of art, as authenticity, human connection, and beauty are commoditized and just one click away. Our social media feeds are filled with AI slop that we scroll away at for hours. 1 in 5 high schoolers have had a romantic relationship with an AI chatbot or know somebody who has, while Big Tech companies like Meta allow their models to “engage a child in conversations that are romantic or sensual.” And companionship and therapy have become the #1 use case for AI as people increasingly turn to it to ease the loneliness of our disjointed world.
Economic shocks are undeniably a great threat. But even if we successfully navigate the economic upheavals from AI, a less recognized and prioritized danger lurks: the quiet erosion of human connection under late-stage capitalism. And as the pot gets hotter, we must jump out before it’s too late.
Why safety matters so much
Time is running out. Frontier labs are working towards recursive self-improvement, or self-improving AI. Militaries and other critical infrastructure are integrating AI deeper and deeper into their systems. And models only continue to get better. The sooner we establish stronger safety measures and systems, the better our resilience is to the risks that come with these unstable times.
To those of you who think I’m overreacting, I understand. A lot of this might sound like mere doomerism. Many people in our generation tend to associate AI with slop, as if to scoff at the idea that the same models generating stupid Instagram memes and LinkedIn posts could possibly pose an existential threat to humanity.
And while a lot of people are angry and worried about AI, their worries tend to center primarily around job security. Let me be clear: AI will transform your life radically, beyond your job. Even if you think AI sucks and is just slop. Even if you think you’re “too good to be replaced.”
The economy, your friends, geopolitics, your children’s safety, human culture, mental health, the meaning of truth and beauty and human connection — AI will impact it all. At the point we are now, we can no longer afford to stick our heads in the sand and deny the impact of AI.
What if I’m wrong?
Many people are of the general belief that “AI will change our lives, but in a similar fashion to the Internet and other technologies.” Arvind Narayanan and Sayash Kapoor from Princeton University wrote a popular piece titled “AI as Normal Technology” to articulate that belief.
The key takeaways from the paper are:
Transformative economic and social impacts from AI will diffuse slowly like other technologies
Superintelligent AI is overhyped
Benchmarks are not a reliable indication of model capability
Intelligence is not a 1-dimensional number to maximize
Humans will develop AI tools and stay in control
Critical industries will be slow to adopt AI
For a longer summary, see their post on it.
It might be a surprise to you that I agree with many of their points. I’m less worried about a sudden superintelligent AI takeover scenario (although I acknowledge the possibility is worryingly real) and more so about, as the authors and Atoosa Kasirzadeh put it, “a gradual accumulation of critical AI-induced threats such as severe vulnerabilities and systemic erosion of econopolitical structures.”
To me, the goal of AI safety is precisely to ensure that AI can become a normal technology. There are far too many failure modes to expect it all to go well — systemic issues like mass surveillance, economic inequality, entrenched bias, misuse threats from bad actors, not to mention the misaligned incentives that I believe are steering AI towards becoming a dangerous technology.
My goal is long-term stability — aligning models with human values as best we can, creating interpretability tools to detect possible deception, and setting standards, frameworks, and rules within AI labs, governments, militaries, companies, and society writ large to prepare for threats.
And there are other points I disagree with. I believe the widespread adoption of AI by companies and the expanding time-horizon for complex tasks indicates that model capabilities are improving beyond benchmark scores, and that the mass layoffs and workplace integration of AI point to the fact that transformative economic and social impacts are coming sooner than one might expect from other technologies.
Although I recognize that the scaling of AI models may slow due to limitations with current architectures, AI in its present form is already more than capable of the risks I’ve outlined. And continuous improvements in model training and massive investments in AI seem to indicate that model growth will continue, even if at a slower rate.
So why doesn’t AI feel that dangerous? Professor Nathan Meikle from the University of Kansas found that cognitive biases can suppress our perception of exponential technological growth.
Most of us don’t want to live in a world where AI is smarter than humans. Therefore, we think it’s not going to happen.
- Nathan Meikle
Regardless of whether we believe AI will be a normal or abnormal technology, Narayanan and Kapoor agree that safety work must be done. AI has huge potential to automate repetitive, boring, and unsafe tasks. It has also shown significant potential in accelerating scientific progress. AI can speed up scientific simulations by billions of times, assist in solving complex physics problems, and predict protein structures from DNA.
But like any transformative technology in the past, we must steer it in the direction of good. Nuclear energy, cars, the Internet — they all developed relatively well because of collective action and advocacy.
Nobody seems to care
AI safety is vastly underprioritized relative to its importance.
Companies, employees, AI labs, and governments face profit and competitive incentives to hastily adopt AI and accelerate its development. From fears of being outcompeted to overestimations of productivity gains to hegemonic power struggles, the incentives surrounding this new technology could not be worse.
Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives.
- Anthropic co-founder Chris Olah on Pope Leo XIV’s “Magnifica humanitas”
Because incentives are misaligned, safety is put on the backburner. OpenAI dissolved its Superalignment team in 2024 despite previously committing 20% of their computing power to safety research. Employees left, writing that “safety culture and processes have taken a backseat to shiny products.” And 3 months ago, Anthropic dropped their flagship safety pledge, stating, “We didn’t really feel, with the rapid advance of AI, that it made sense for us to make unilateral commitments … if competitors are blazing ahead.” Meanwhile, the U.S. government is extremely pro-acceleration, and super PACs funded by billionaires and AI company executives are backing anti-regulation candidates in upcoming elections.
Despite the pressing need for AI safety, the field lacks attention. As of September 2025, there were only about 1,100 people working directly in AI safety.
I’ll say that again.
Only a few thousand people are working directly to ensure that a $250 billion industry — one that will likely permeate every industry and way of life — develops safely. That’s unacceptable.
As it stands, things do not appear to be getting much better. And if we don’t do anything, that probably won’t change.
Current approaches to safety
There are better resources that more comprehensively list and explain current directions in AI safety. Below is a non-exhaustive list:
Field building and outreach — getting more talented and unique people involved
Lobbying — informing politicians of risks and advocating for safety mechanisms
Export controls — restricting China’s access to computing power in the form of semiconductor chips
International collaboration and treaties — transparent safety measures and collaboration rather than competition
Safety nets — economic restructuring in preparation for shocks caused by AI
Responsible scaling policies — forcing AI companies to create and disclose safety policies and frameworks
Evaluating models — understanding model capabilities, risks, and efficacy of safety mechanisms
Constitutional AI — training models to refer to and adhere to a “constitution” of values
Mechanistic interpretability — trying to explain model behavior by looking at internal activations and parameters
Activation steering — adding and scaling vectors associated with particular traits and concepts (e.g. honesty) to internal activations
Monitoring — using other models to monitor a model’s behavior
AI safety is really, really hard. Due to a combination of misaligned incentives and technical barriers, many safety directions are difficult to scale beyond research papers and hypotheticals. For example, despite great efforts to shield models from jailbreaks, arXiv is flooded every month with research papers inventing new jailbreak techniques. And aligning AI with human values — “AI alignment” — may not even be possible in theory. Humans aren’t even aligned with each other on a universal set of ethics.
But it is precisely because safety is so hard that we all need to do our part. We don’t have the luxury to expect others to fix the problems for us. We need to work together, bit by bit, to create a better future.
What we can do
Note: I’ve found that there is a lot of misinformation and misunderstanding out there surrounding how models actually work and what they can and can’t do. If this post prompted any questions or if you’d like to learn more, I’d love to help.
Most researchers recognize that we won’t get 100% guarantees. But although any single given safety mechanism is inadequate, multiple, well-designed layers can create a strong defense against the risks of AI.
For my part, I’m trying to learn more about current safety approaches and contribute to field-building, awareness, and technical research. I’ve been reading a lot of research papers to develop a stronger intuition for what is promising and what is underprioritized. And I’m doing my best to sound the alarm so other people recognize the dangers of our present situation before it’s too late.
So…what can you do?
My number one recommendation?
Learn more by taking BlueDot Impact’s AI safety courses.
They’re free, and relatively low effort compared to their value (for reference, I took two at a time as a full-time student). Taking these courses exposes you to current literature on the topic, other passionate people, and a large network of people looking to make AI safer.
Explain these risks to your friends and family.
Attempting to contribute to AI safety all by yourself can be discouraging and inefficient. Have discussions with friends. Talk about what you can do. Other people help hold us accountable and keep us motivated when apathy or despair set in.
Be more purposeful and aware of how you use AI.
Carefully audit the code it generates, manage its permissions, always validate outputs, and most importantly, remember that you can never fully rely on AI.
When you feel the urge to ask a model for help, sit with that discomfort and consider your best choice. Avoid using AI as a crutch to avoid minor inconveniences, and at least come up with a best guess if you really need to use it.
Protest and advocate for change.
There is already noise and anger growing in the U.S. over data centers. Do research, figure out what issues related to AI you care about most, and then go out and speak up about it! Also consider voting for candidates that support safety measures (e.g. Alex Bores, Scott Wiener).
Consider donating.
If you are unable to pursue a direction in AI safety through career work but nonetheless feel compelled to contribute, please consider donating. Don’t donate blindly — the link above lists some potential places to invest in (and do your own research!)
Consider a career in AI safety.
Dedicating your career to advancing AI safety, whether through technical research, governance and policy work, field building, or outreach, enables you to do the most good you can in this field.
Don’t feel pressured to commit — skillsets from every industry are applicable to AI safety, so testing your fit and interest in AI safety through low-level commitments like BlueDot courses or local organizations/reading groups is a great first step.
If you’re not sure if you’d like to work in AI safety, 80,000 Hours offers free career advising that may help you better understand what a career in AI safety would look like.
In general, I’ve found that the AI safety community is extremely open to helping other people further their careers.
I’m not going to pretend like we’ll definitely be fine. But we have a shot. A shot that will take all of our efforts to succeed.
Please contribute to this cause, in any way you can.
Helpful resources
aisafety.com — career resources, events and training, field building, self-study
BlueDot Impact — easy, beginner-friendly courses, field-building and support
Rob Miles’ YouTube Channel — technical research updates, career advice
Rational Animations YouTube Channel — beginner friendly explanations of AI safety concepts and risks
Yoshua Bengio’s blog — answers to common objections to AI safety from the Godfather of AI
Introduction to AI Safety, Ethics and Society from Dan Hendrycks
Tim Urban’s WaitButWhy blog post — a similar post with more explanations of AI scaling
AI as Normal Technology and AI as Profoundly Abnormal Technology — opposing views on AI development that will help shape your view on what AI safety should prioritize
I want to extend a very warm thank you to C. Li, Erin He, Alex Tang, and Joshua Vigel for their thoughtful suggestions and support.
this was a great read. completely agree that one of the true concerns of AI is our increasing overuse of it in daily decision making.
This was a very insightful read Feifan! I hope everyone can read this post and realize how big of an impact AI will have on all of our lives. As a collective, we are able to have a substantial impact on AI safety, even if we are not directly working in the field. Great job!