Can LLMs be creative?

How far out of distribution can modern AI systems go?

Audio version available here.

A few months ago I was out to dinner with a childhood friend. He is not an AI specialist, but he has a degree in computer science. At some point the topic of AI came up, and he shared with me why he thought current systems will never be truly intelligent: all they do is combine the data in their training set, and thus can never possess true creativity. His example was one repeated often. Early image generators seemed to be incapable of creating an image of a full glass of wine, presumably because all the pictures you see online (and hence all the images they were trained on) depict half-full glasses. When I tried in late June 2025, I had no problem using an AI image generator to create a picture of a full wine glass. However, the question is still a good one: Can current AI systems ever be truly creative, and produce something completely “out of distribution” (i.e. not found in their training data)?

In this post I’m going to focus primarily on one particular kind of creativity: creative problem solving. The question here is, “to what extent can AIs think ‘outside the box’, and come up with novel solutions to problems they’ve never seen before?” This is intimately related to whether or not we can ever develop human-level Artificial General Intelligence (AGI), a question that is driving many of today’s big-tech efforts.

When ChatGPT first entered the public consciousness it largely lacked any ability to solve problems it hadn’t seen before. The first version of ChatGPT was based on a model called “GPT 3.5”, which was created by showing it many, many language examples, and having it “learn” only from those examples how to create plausible sounding responses to questions. This is a classic example of “Supervised Learning,” and is a common approach in many areas of Machine Learning (not just neural networks like ChatGPT).

An alternate approach to Machine Learning is called “Reinforcement Learning,” in which a machine interacts with an “environment” and “learns” from positive and negative rewards it receives from those interactions. A classic example of Reinforcement Learning is an algorithm that learns to play a video game: the game is the environment, a change in point score is the reward.

Here’s all you need to remember: In Supervised Learning (SL), models “learn from example.” In Reinforcement Learning (RL), models “learn from experience.”

Reinforcement learning demonstrated that it can lead to creative problem-solving ability several years ago. The most famous example is from a team of researchers at Google who trained an RL model to play the ancient game of Go. Go is a board game commonly played in Asia. It is older than chess, played by more people worldwide, and is significantly more difficult for a computer to master. In 2016 Google’s AlphaGo won a 5 game match against Lee Sedol, who was thought at the time to be the best Go player in the world. The most referenced moment of that tournament is the 37th move of the 2nd game, when AlphaGo did something that initially seemed like a mistake. It broke all common rules-of-thumb, and no expert would have done it. It was so outside-the-box that there was no way for such a move to be in its training data. But AlphaGo didn’t just learn how to play from its training data: it also learned by playing against itself many thousands of times, and learning on its own what a good move and what a bad move was. Move 37 in game 2 turned out to be the pivotal moment of that game, leading directly to AlphaGo’s victory. No Go expert would hesitate to call that move “creative.”

It’s important to realize that RL doesn’t mean the computer just tries thousands of random moves and picks the best one. A game like Go is just too complicated to play out every possible game. There is some randomness, but mostly in the training phase when the computer is playing against itself. During play against a human the computer uses the strategies it learned during training to select moves judiciously.

Breakthroughs in RL development like AlphaGo paved the way for the “reasoning” LLMs that have been developed in the last year. These models differ from the original ChatGPT in that RL has been added as an important step in their training procedure (see Technical Footnote below). This has been most effective in domains where there is a clear reward signal. For example, in solving a math problem (either with calculation or higher order logic) or in writing computer code, there are often clear “right” and “wrong” answers. Finding good reward signals in other domains is an ongoing line of research, which the AI labs are making rapid progress on.

These reasoning systems have stunned many with their ability to be “creative.” Scientific American recently featured a story about a meeting of Mathematicians in Berkeley, California, whose sole purpose was to come up with challenging, novel problems for an LLM. After the meeting Ken Ono, a Mathematician from the University of Virginia, was quoted as saying:

“I have colleagues who literally said these models are approaching mathematical genius. … I was not prepared to be contending with an LLM like this, I’ve never seen that kind of reasoning before in models”. —Ken Ono

Last week both Google and OpenAI announced that their unreleased next-gen models achieved gold-medal level performance on the 2025 International Math Olympiad (IMO), an achievement that stunned even AI experts. Two days later a pair of researchers from UCLA announced similar performance on the IMO 2025 questions from Google’s currently available Gemini 2.5 pro model. Although the IMO is a competition for high-school students, the problems on it are difficult enough to stump many PhD mathematicians. Answers are complex proofs that require sophisticated logical reasoning, not calculations. It’s almost certain that none of the problems appear anywhere in the AI models’ training data.

There’s a lot to be skeptical of in these accomplishments. For one thing, it’s not clear if mathematical prowess translates to the ability to solve novel problems in other domains. However, what these achievements demonstrate is that RL is certainly capable of imbuing AI with creative problem solving abilities. The only ingredient necessary is a clear reward signal. Those reward signals can come from many places, not just “right” or “wrong” steps in solving a math problem. There is a lot of current research in finding good reward signals in business applications, science, technology, medicine, etc.

Perhaps a more serious critique is that even within the domain of mathematics, solving problems is often not the most creative act. It takes much more creativity to come up with good problems in the first place! While the achievements of all the IMO medalists (both human and non-human) are certainly impressive, what’s truly amazing is the panel of mathematicians who are able to come up with genuinely new problems that require that kind of creative problem solving ability. Many have argued that asking the right questions is what pushes any domain in new directions. It’s not clear when, or if, AI will be able to do that.

Finally, creative problem solving is just one kind of creativity. There are many others. Artistic creativity, for example, is more than just making pictures no one has made before. Musical creativity is more than just stringing notes together in new sequences. People recognized as “creative geniuses” push their domains in new ways that are deeply human. Almost by definition, no Artificial Intelligence will ever be able to do that. Humans will always appreciate human-created art, music, literature, visual arts, performance, etc. And I expect humans will always be the ones that ask the questions which push those genres in new directions, raising humanity to new heights.

Technical Footnote: When ChatGPT was first released, there was some RL involved in the training process. This was called RLHF, or “Reinforcement Learning from Human Feedback”, where the reward signal for good/bad responses was derived from feedback provided by human evaluators. This is very different from the more contemporary use of RL called “Reinforcement Learning with Verifiable Rewards,” where the reward signal for reasoning steps in a “chain of thoughts” comes from more objective data.