AI reasoning ability is only 0.37% of a human's.
Two weeks ago,
a new test was quietly launched.
It didn't make any major headlines,
but within Silicon Valley's inner circles,
it caused many sleepless nights.
Because the results of this test show
that today's most powerful AI,
when faced with entirely new environments, has only 0.37% of human reasoning ability.
Not 37%,
but 0.37%.
Hello,
everyone.
I'm Wang Lijie.
Today I want to talk to you about a test called ARC-AGI-2,
published on March 25, 2026.
If you follow AI news,
you know how crazy the pace of AI benchmarks has been in the last two years.
One day a model tops a leaderboard;
the next day, another model pushes it off.
Every few weeks, someone announces AI has surpassed humans in another exam.
Amidst this atmosphere of constant triumph,
a test suddenly appears saying AI is only at 0.37% of human level.
It sounds almost like a joke,
but it isn't.
It might be the most important "reality check" in the AI field in years.
The creator of this test is François Chollet,
a French AI researcher.
He worked at Google for over nine years
before leaving at the end of 2024 to start his own venture.
He's no outsider;
on the contrary,
he is the author of Keras, the deep learning framework
used by thousands of AI engineers worldwide
to build neural networks.
In other words,
he's a builder of AI tools.
Yet, it was this very "insider"
who published a paper in 2019
titled "On the Measure of Intelligence."
The core argument of this paper can be summarized in one sentence:
We have been measuring AI intelligence the wrong way.
What does that mean?
Think about it.
All those AI report cards we've seen lately—
exam scores, leaderboard rankings—
what exactly are these tests measuring?
They measure if AI can solve known types of problems,
like doing math, writing code, or answering trivia.
AI is indeed getting better at these,
even surpassing humans in some cases.
But Chollet says
that this isn't intelligence at all—
it's memorization.
He makes a distinction
using a classic psychological concept: fluid intelligence vs. crystallized intelligence.
Crystallized intelligence is your accumulated knowledge and skills.
Like speaking English,
remembering the Pythagorean theorem,
or knowing that Beijing is the capital of China.
These are things you've learned, memorized, and can recall at any time.
Fluid intelligence is completely different.
It's your ability to spontaneously find a solution
when faced with an unfamiliar, brand-new problem.
As a simple example,
imagine being dropped into a completely strange city.
You don't speak the language,
have no phone signal,
and no map.
Can you find your way back to your hotel?
What would you do?
You might observe local landmarks,
pick a direction, and try it out.
If it's wrong, you adjust based on new clues.
Throughout this process, you aren't using pre-stored knowledge,
but an ability for real-time perception, probing, feedback, and adjustment.
That is fluid intelligence.
Every normal person has it.
[MISSING]
And it's highly efficient.
Chollet's core argument is that today’s AI,
almost all of its amazing performance comes from crystallized intelligence.
They have all the public text in human history as training data.
Their "knowledge base" has long since surpassed any individual human.
Ask it any known question,
and it can give you a decent-looking answer.
But that isn't intelligence.
That's memory plus pattern matching.
What is true intelligence?
It's when you face an unprecedented situation,
and the efficiency with which you learn and adapt.
It's not about how much you know,
but when you don't know,
how fast you can figure it out.
To prove this point,
he designed the ARC-AGI test.
The first version was called ARC-AGI-1,
a series of abstract pattern transformation problems.
You are given a few sets of input-output pattern examples,
you must infer the underlying rules,
and then apply those rules to a new input.
For example,
you see three sets of examples,
each showing a blue square moving to a certain position.
You need to find the movement rule
and predict where the next square will move.
The questions aren't hard;
an average person can figure them out at a glance.
But the key is that the rule for each problem is unique.
You can't memorize answers in advance.
It tests whether you can "get it" on the spot.
The results were very interesting.
Initially, AI performed poorly on this test,
scoring only between 0% and 30%.
Then AI companies began to focus their efforts on cracking this test.
By late 2024,
OpenAI released a reasoning model called o3.
Using its highest compute mode,
consuming 172 times more compute than the standard version,
it achieved a score of 87.5%.
Suddenly, it surpassed the 85% human average.
Silicon Valley went wild.
Many cheered,
saying the dawn of AGI had arrived.
But few noticed another number.
The computational cost for this test was astronomical.
Initial estimates were about $3,000 per problem,
totaling over $1 million for 400 problems.
Later, someone recalculated it
and believed the real cost could be as high as $30,000 per problem,
totaling over $10 million.
Regardless of which figure is correct,
it's terrifying.
Meanwhile, an average human doing the same 400 problems
uses energy roughly equivalent to eating one lunch.
Human labor costs about $5 per problem.
You spend millions or even tens of millions of dollars to tie
with someone who just ate a sandwich.
Is that a breakthrough?
More importantly,
when researchers analyzed o3’s process, they found
it didn't actually "understand" the rules.
It used brute-force search,
probing through thousands of possible reasoning paths,
backtracking, re-selecting,
until it happened to find a path that worked.
That's not thinking;
that's exhaustive search.
It's like trying to find the code to a combination lock,
not by understanding the logic behind the code,
but by trying from 0000 to 9999.
One of them will eventually open it.
You did open the lock,
but do you really "understand" the lock?
Seeing these results,
Chollet wasn't discouraged;
instead, he doubled down.
He first launched ARC-AGI-2,
significantly increasing the difficulty of the problems.
AI scores plummeted from 93% to 68.8%.
Then in March this year,
he released ARC-AGI-3.
This version completely changed the rules of the game.
ARC-AGI-3 is no longer just static pattern puzzles;
it features interactive environments.
Think of them as miniature video game worlds.
Each world is handcrafted by human game designers,
with hundreds of completely different environments.
But these games come with no manuals,
no rule introductions,
and they don't even tell you the objective.
You are thrown into a strange world
and must click and use trial and error,
observing the changes caused by every action.
Then you must figure out three things: how this world works,
what the goal is,
and what counts as winning.
And it's not just about whether you pass;
it also tracks how many steps you take.
It calculates your learning efficiency—
the speed at which you turn environmental info into effective strategies.
This is Chollet's 2019 definition of intelligence:
skill-acquisition efficiency over unknown tasks.
This is the true test of fluid intelligence.
It's not giving you a problem to solve;
it's throwing you into a totally unknown environment
to see if you can figure out what's happening from scratch.
Humans scored 100% on this test.
Every single human participant successfully solved every environment.
As for the world's most powerful AIs?
Google's Gemini 3.1 scored 0.
37%.
OpenAI's GPT-5.4 scored 0.
26%.
Anthropic's Claude
Opus 4.6 scored
0.25%.
Most absurdly, Grok-4.20
scored a flat zero.
Listen carefully.
It's not just that AI is slightly worse than humans;
it's not a gap of 10 or 20 percent.
It's a gap of two to three orders of magnitude.
Humans: 100%.
Strongest AI: 0.37%.
If humans are standing at the mountain peak,
AI hasn't even found the entrance at the foot of the mountain.
Some might say,
"That's not fair."
These interactive environments are designed for humans.
AI doesn't have hands
or eyes.
How can it operate?
But the thing is,
these environments run entirely on a computer screen,
consisting of blocks and colors on a grid.
The AI can see the screen and execute actions;
it has the exact same input and interface as a human.
It doesn't lack senses,
computing power,
or data.
What it lacks
is understanding.
Even more intriguing is a detail from the preview phase:
the top performer wasn't a trillion-parameter LLM.
It was a relatively simple system built with
reinforcement learning and graph search.
It scored 12.58%,
an order of magnitude higher than all the LLMs.
What does this show?
It shows that the LLM approach—
learning statistics and patterns from massive text—
is move-for-move the wrong methodology
when facing truly novel problems.
Language is not the sum total of intelligence.
Feeding the entire internet's text into a system
won't teach it to adapt through trial and error in a new environment.
This reminds me of a very vivid metaphor.
You know the story of AlphaGo.
AI can defeat world champions at Go.
Although Go has endless variations,
the rules are fixed, known, and explained from day one.
AI can achieve extreme optimization within this space of known rules,
even surpassing humans.
But what ARC-AGI-3 does is completely different.
It's like being thrown into a board game you've never seen.
No one tells you how the pieces move,
or what counts as winning.
You have to play a few rounds and figure it out yourself.
Humans don't find this very difficult;
we get the hang of it after a few tries.
But AI?
It becomes almost completely paralyzed.
Consider a scenario from daily life.
You visit a friend's house,
and their toilet has a flush mechanism you've never seen—no button,
no pull cord,
but some strange rotating device.
You might pause for three seconds,
feel it out,
give it a turn,
and the water flushes.
You never learned how to use that specific device,
but you just "got it."
You take this ability for granted.
But for today's most powerful AI,
this is the real challenge.
It can write poetry, prove math theorems, and generate code,
yet it can't handle an unfamiliar flush button.
So, what do these experimental results really mean?
If you watched my previous video on Penrose's theory of consciousness,
you might already be connecting the dots.
Starting from mathematics,
Penrose used Gödel's Incompleteness Theorem to argue a startling conclusion:
The human mind contains something "non-computable."
There are things we can do
that in principle, no algorithm can achieve.
That was a purely theoretical deduction,
whereas ARC-AGI-3 provides experimental
evidence.
It doesn't talk about consciousness,
the soul,
or quantum mechanics.
It simply designs a simple test: in a brand new environment,
can you learn to adapt from scratch?
The result? All humans passed,
while the AI failed across the board.
A theoretical hammer,
and an experimental hammer,
hitting from two completely different directions,
strike the same wall: there is something in human intelligence
that cannot be replicated by stacking data, parameters, or computing power.
This comparison is harsh,
but it's also very real.
We are so dazzled by AI's fluency in language and logic
that we forget one fact: fluency does not equal understanding.
A parrot can perfectly mimic every word you say,
but it doesn't know what those words mean.
Large language models are far more complex than parrots,
but ARC-AGI-3 is asking the same kind of question.
When you strip away all pre-trained knowledge and learned patterns,
leaving only the ability to "figure it out on the fly when facing the unknown,"
what does AI have left?
The answer is: almost nothing.
I want to emphasize: I'm not saying AI will never catch up.
Technology is advancing,
architectures are innovating,
and perhaps someone will find a breakthrough next year.
Chollet himself has set up a competition prize of $2 million
for anyone who can reach human performance
on ARC-AGI-3.
But the gap revealed by ARC-AGI-3 today
is not a simple quantitative difference.
It's not that AI will catch up just by continuing to train
or expanding its parameters.
It exposes a qualitative difference: the entire current AI paradigm,
from LLMs to reasoning-enhanced models,
lacks a fundamental ability
when facing true, absolute unknowns.
What is this ability exactly?
Chollet gives a technical answer: fluid intelligence.
The ability to efficiently acquire skills in a completely new environment.
But if you dig a step deeper,
this answer just renames the problem.
The real question is: why are humans born with this ability?
If you put a three-year-old in a room they've never been in,
without a teacher,
or a manual,
within minutes, they'll figure out how the doorknob works,
which button turns on the lights, and where the blocks go.
They don't need to train 100,000 times.
They don't need to search through every possible combination of actions.
They take one look,
and just "get it."
What exactly is this "getting it"?
It's not computation.
Because if it were,
AI—being a trillion times faster than humans—
shouldn't be worse at it.
It's not a knowledge base,
since a three-year-old has almost zero knowledge.
Nor is it an instinctive response granted by evolution,
since doorknobs and blocks weren't part of our evolutionary environment.
It seems to be something more fundamental—
an ability to instantly organize fragmented perceptions
into a "meaningful whole."
The ability to see "what this thing is for"
without being taught.
Cognitive scientists call this "Gestalt perception."
But that's just labeling the mystery,
not solving it.
This is what's truly unsettling about
ARC-AGI-3.
It's not just saying AI isn't strong enough
and needs further optimization.
It asks a deeper question:
What is "intelligence," anyway?
We've always thought of intelligence as information processing: data in,
process it,
answer out.
By that definition,
AI should have outsmarted us long ago.
But ARC-AGI-3 shows that,
when facing the truly unknown,
the human "biological computer," processing only 10 bits per second
on 20 watts of power,
demolishes silicon systems built with millions in electricity costs.
Perhaps
intelligence was never about information processing.
Perhaps "understanding"
cannot be attained by simply adding more computing power.
Perhaps the moment a human understands something they've never seen,
what occurs isn't a calculation,
but a physical event—
something current computer architectures simply cannot simulate.
ARC-AGI-3 doesn't answer the question,
but with that cold 0.37% figure,
it pins the issue firmly to the table,
making it impossible for anyone to ignore.
Do you think AI can eventually bridge this gap?
Or is the gap itself telling us
that we've underestimated "understanding" from the start?
If you have any thoughts,
I'd love to hear them.
I am Leo Wang.
See you next time.