Why AGI is impossible
While AI will be able to do a lot more, it'll never do everything
Last week I wrote, “I’m a believer that AGI is possible”, where “AGI is the ability of models to do anything a human might do at a computer terminal.” I shouldn’t have been quite so glib. With that broad of a definition, AGI is not possible, and in this post I’ll explain why. AI will be able to handle an increasingly large portion of the tasks that humans do, but it will never be capable of doing all of them.
There’s no real debate that LLMs are able to do specific things better than humans. However, the “G” in AGI stands for “General,” indicating a level of intelligence that meets or exceeds human ability at a very wide variety of tasks. But “general” shouldn’t be conflated with “all,” which many authors (including myself!) do. In fact, its not hard to nail down precisely which tasks we can hope for human (or even super-human) level AI performance, and which tasks we can’t: the key criterion is professional consensus. To understand, let’s first talk about where LLMs actually get their intelligence from.
There are two basic phases in LLM development: pre-training and reinforcement learning (RL). In pre-training, the nascent LLM is fed a lot of information (text, pictures, audio, video) and from that it learns statistical patterns. An LLM that has only been pre-trained is then capable of mimicking those patterns. Some have called them “Stochastic Parrots” for this very reason. However, pre-training is only the first phase in modern LLM development.
The RL phase is arguably where much more of the intelligence is imbued. In RL, models are given rewards for reaching a goal, and they “learn” correct behavior based on accumulated rewards. What’s important for this to work is the consistency of those goals/rewards. If they are always told that 1+1=2 then they can “learn” that fact quickly. If, on the other hand, they’re told 1+1=2 half the time, and 1+1=3 the other half, they’ll have a much harder time learning anything.
This is precisely why AI-assisted coding has become so good in recent months. Coding has a very stable, consistent reward signal. The code either does what it’s supposed to do or it doesn’t. There are also a lot of standard practices in writing and structuring code. Many claim the newest coding models (Anthropic’s Claude Opus 4.6 and OpenAI’s GPT 5.4) are now at least as good as the average software engineer, and supposedly Anthropic’s new (unreleased) “Mythos” model is significantly better. The fact that the reward signal in coding is so strong is also what makes super-human performance possible: the models aren’t just trained on human generated code, they can “discover” new ways of coding that lead to correct solutions on their own.
Mathematics is another domain with a very clear reward signal: a mathematical proof is either correct or it is not. For a variety of reasons it’s harder to train AI to do math reliably than to code, but I’m very confident that eventually the models will surpass human level performance in this domain as well. Their mathematical abilities have already come a very long way in a very short amount of time.
Even in domains where the correctness of the goal is more subjective, RL can be extremely effective, as long as there is consistency. For example, the entire medical profession is predicated on the idea that most doctors will be able to correctly diagnose most ailments, which makes diagnosis a fairly consistent task. This is thus an area where we can expect AI to become as good as most humans (and in fact, we’re seeing exactly that in the latest models). Similarly, many business practices follow very standard workflows, so it is likely that AI will master those as well (if it hasn’t already).
So what are areas where AI will always struggle? Any field where there is no general consensus on the products of that discipline. Consider the related practices of Art versus Design. In the fine art world there are no universally agreed on standards for what makes good art vs bad art. That means there will never be a consistent reward signal, and so we can expect AIs will always struggle to achieve a human-level ability to make fine art. Design, on the other hand, is somewhat less subjective. There are well established principles of good design vs bad design, and as a result AI-assisted design tools have become quite effective.
What happens when there is an inconsistent reward signal? The model essentially learns to produce the average. This is why so many have critiqued AI-generated writing as being mediocre. That can still be enough to be incredibly disruptive: an average writer is, by definition, a better writer than a large swath of the population. Model developers can also improve outputs somewhat by hiring humans to rate writing samples, and feeding those ratings to the models during training. That can effectively raise the level of AI-generated writing from, say, a “C” level to a “B” level. However, with current training paradigms it’ll be very hard for them to ever get to an “A+,” which is what AGI might imply.
We talk about AGI as a milestone. For many, many reasons it’s important to know how close current models are to that level of intelligence. However, milestones are only useful if they are achievable. If we define AGI in a way that includes all human intellectual tasks, the models will never get there. There are just too many areas with inconsistent reward signals. A more useful definition of AGI should only apply to those tasks for which there is a consensus among professionals on what is “good” versus “bad” performance. With such a definition in hand, we can have a much more meaningful discussion about how far current models are from AGI.
Note that I’m advocating here for a nuanced definition of AGI that talks about task competency, not the ability to replace entire professions. Most professions involve many tasks, and very likely the future of those professions relies on the balance between which of those have some kind of consistency when done well, and which don’t. As we move closer to AGI, jobs will shift so that most professional human tasks require highly individualized human judgement. If you want to keep ahead of this technology as it develops, that’s the professional shift you should be thinking about.
App of the Week
I continue to make apps with AI-assisted coding tools. These apps run the gamut of things I find personally useful to things I think will be entertaining, educational, or useful for research. One of the reasons I’m so bullish on AI, despite all of my worries about it, is that I wouldn’t have been able to make any of these apps without this technology.
Recently I’ve given a few workshops on how to use AI-coding tools to make these kinds of applications. (Feel free to reach out if you’re interested in hosting such a workshop!) During those I often solicit app suggestions from the audience, and walk them through how to start the process of creating it.
In one of the last workshops I gave, an audience member suggested an app that shows users how to unfold a polyhedron. I was excited about this; I’d been thinking about making an app to manipulate polyhedra for a while, but it hadn’t occurred to me to incorporate an unfolding action. To be honest, I was a little nervous because I didn’t even know if this was something the LLMs would be capable of creating, and it’s never great to do a public demo that fails (although sometimes the way in which it fails can be very instructive). In the end, though, everything worked so well that even I was surprised! After the workshop I added several features, but much of the core application was really built during that session. Click the image below to see a short video of it in action. Click here to play with it.
Recently John Bowers, a mathematician who studies polyhedra unfolding, told me how much this little app inspired him. He used AI coding assistance to make a more sophisticated version that does slightly different stuff but is more suited to his research agenda. I highly recommend checking that out as well, here.
That’s exactly the promise of AI! Being able to quickly share and iterate on ideas to aid in research, education, creativity, productivity and entertainment can be an incredible boon to human flourishing. Of course, it’s not without its costs, but people don’t weigh those against these kinds of benefits of AI enough.
Coffee with Digital Trailblazers!
I’m honored to be speaking alongside an amazing panel of experts at this free event on “AI Coding Competencies,” April 17th at 8am PST/11am EST. Click the image below to learn more. Hope you can make it!
David Bachman is a professor of Mathematics, Data Science, and Computer Science. He writes about AI and its real-world impacts. To learn more about his academic work, mathematical art, or AI speaking, consulting, and curriculum development, visit davidbachmandesign.com.