What is perplexity in AI? OpenAI publishes its scores, from GPT-3.5's 0.986 to an internal model's 0.732

OpenAI logo with the text: Publishes its models perplexity scores

Key points

  • OpenAI publishes the perplexity scores it uses to track its models
  • OpenAI's unreleased internal model scores 0.732, its lowest yet
  • OpenAI's chief scientist proposes industry limits tied to the metric

OpenAI published the perplexity scores it uses to track progress in training its AI models, and Chief Scientist Jakub Pachocki called for developers to use comparable measurements to set shared safety standards.

In a Nov. 22 post, Pachocki proposed that labs agree on development limits tied to perplexity and other measures. OpenAI's latest unreleased model recorded its lowest score yet on the company's test.

Publishing the scores "gives competitors a clearer picture of where we stand," Pachocki wrote, but he said the need for shared safety measures outweighed that concern.

Perplexity measures next-token prediction

Perplexity measures how well a model predicts the next token, a word or piece of a word, on average. A lower number means better predictions. OpenAI reports the logarithm of perplexity, which is also called cross-entropy loss.

OpenAI measures it on held-out data, text the model wasn't trained on, taken from OpenAI's internal codebase. The score tracks progress in pretraining, the first stage of training on a large body of text, rather than later steps such as fine-tuning.

The metric is unrelated to Perplexity, the AI search startup.

OpenAI's scores fell faster in 2026

The chart in OpenAI's post shows the score for each model, with labels for some of the abilities OpenAI says each one reached.

ModelDate on chartLog-perplexityOpenAI's label
GPT-3.5March 20220.986Fluent conversation
GPT-4March 20230.913Benefits from thinking before answering
GPT-4 TurboNovember 20230.909
GPT-4o / o1May 20240.904
GPT-4.5February 20250.876
GPT-4.1 / o3April 20250.902
GPT-5 / 5.1August 20250.893
GPT-5.2December 20250.888
GPT-5.3-Codex / 5.4February 20260.871Useful coder
GPT-5.5April 20260.848
GPT-5.6 SolJuly 20260.828Cybersecurity expert
GPT-6 AstraSeptember 20260.783
GPT-6.2 AstraNovember 20260.761Solved a Millennium Prize math problem
Internal model (unreleased)November 20260.732

Source: OpenAI, chart data in its Nov. 22, 2026 post. Lower is better. Scores are measured on held-out data from OpenAI's internal codebase.

The scores come from a fixed snapshot of that codebase. They aren't a universal intelligence score and can't be compared directly with other labs' numbers, and OpenAI cautions that the choice of dataset affects how informative the measure is for different models.

The score fell 0.110 from GPT-3.5 to GPT-4.5 over nearly three years, and 0.156 from GPT-5.2 to the internal model in about 11 months, by our calculations. OpenAI attributes the steep acceleration in 2026 mostly to long-term research and computing investments and only in a smaller part to automation. For the most recent models, OpenAI has "already seen meaningful design contributions coming from AI," Pachocki wrote.

The chart doesn't say which Millennium Prize problem GPT-6.2 Astra solved. OpenAI said in September that a model had resolved the Navier-Stokes problem, one of the seven Millennium Prize Problems. Scientific American reported that the result relied on an external force that the Clay Mathematics Institute's formulation allows, which many experts consider a loophole. On Nov. 20, Tristan Buckmaster of New York University posted an AI-aided proof with Anthropic's Levent Alpöge that the two say closes that loophole and covers the version without an external force, the magazine reported.

Pachocki expects new capabilities near 0.68 to 0.70

Pachocki wrote that once OpenAI's models reached losses below 0.8, it saw "the emergence of transformative capabilities in fields like cybersecurity and mathematics." A model with a score of about 0.76 "started producing groundbreaking results" on open math problems, he wrote.

"Extrapolating from recent progress, we should expect further transformative capabilities as we enter the ~0.68-0.7 log-perplexity window," Pachocki wrote. "I don't know what those capabilities will be, and we may not be testing for them today." OpenAI's unreleased internal model is at 0.732.

He also wrote that AI could speed up research on pretraining itself, and that "as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements" to the one OpenAI saw in math.

Pachocki proposes limits that labs agree not to cross

Pachocki proposed that AI developers agree on "common limits, based on perplexity and other measurements," that they would not cross until their ability to understand and monitor AI behavior reaches a concrete milestone. His example of such a milestone is a successor to chain-of-thought monitors, tools that read the reasoning a model writes out before it answers.

He listed problems with the metric. Different labs measure perplexity on different datasets, so their numbers can't be compared directly yet. Perplexity doesn't capture gains from chain-of-thought reasoning, which Pachocki wrote "leads to a drastic increase in capabilities." And no one knows how low the score can go.

Pachocki called for "extreme caution" in an earlier essay in September, as we reported at the time. The new post comes two days after Alpöge posted a claimed proof of the Riemann hypothesis developed with Claude. The Lean and Mathlib communities have certified its formal proof, while mathematicians are still working to understand the argument.

Frequently asked questions

What is perplexity in AI?

Perplexity measures how well an AI model predicts the next token, a word or piece of a word, on average. A lower number means better predictions. OpenAI reports the logarithm of perplexity, also called cross-entropy loss, and uses it to track progress in pretraining its models.

What perplexity scores did OpenAI publish?

In a November 22, 2026 post, OpenAI published log-perplexity scores measured on held-out data from its internal codebase. GPT-3.5 scored 0.986, GPT-4 scored 0.913, GPT-6.2 Astra scored 0.761, and an unreleased internal model scored 0.732. Lower is better.

What does OpenAI expect at lower perplexity scores?

OpenAI Chief Scientist Jakub Pachocki wrote that OpenAI should expect further transformative capabilities as its models enter the roughly 0.68 to 0.7 log-perplexity window, and that he doesn't know what those capabilities will be. He proposed that AI developers agree on common limits, based on perplexity and other measurements, that they would not cross until their ability to monitor AI behavior reaches a concrete milestone.

Is AI perplexity related to the company Perplexity?

No. Perplexity the metric is a standard measure of how well a language model predicts text. It is unrelated to Perplexity, the AI search startup.

More coverage

Latest stories

Dennis Singleton
Dennis Singleton

Dennis Singleton was born in Australia and later moved to the United States. He has spent years following the markets, but what keeps his attention is how AI is built. He writes about the companies behind the technology, from semiconductor designers and advanced packaging to photonics, memory, networking, and the hardware powering modern AI. His approach starts with filings, earnings, and industry research, then translates the important details into clear, straightforward analysis without unnecessary hype.