The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, “Reasoning” models, and Agentic frameworks.
ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels.
Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what’s next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations.
You can try the tasks yourself here: https://arcprize.org/arc-agi/3
Here is the current leaderboard for ARC-AGI 3, using state of the art models
- OpenAI GPT-5.4 High - 0.3% success rate at $5.2K
- Google Gemini 3.1 Pro - 0.2% success rate at $2.2K
- Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K
- xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K.
(Logarithmic cost on the horizontal axis. Note that the vertical scale goes from 0% to 3% in this graph. If human scores were included, they would be at 100%, at the cost of approximately $250.)
https://arcprize.org/leaderboard
Technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf
In order for an environment to be included in ARC-AGI-3, it needs to pass the minimum “easy for humans” threshold. Each environment was attempted by 10 people. Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Many environments were solved by six or more people. As a reminder, an environment is considered solved only if the test taker was able to complete all levels, upon seeing the environment for the very first time. As such, all ARC-AGI-3 environments are verified to be 100% solvable by humans with no prior task-specific training
As a psychiatrist, I have a theory about what’s missing in AI. First, it lacks childhood dependency and attachments. Second, it struggles to overcome repeated pain and suffering. Third, it lacks regular eating and restroom breaks. Fourth, it struggles to accept loss in everyday situations. Finally, it lacks the concept of our inevitable death. Without these nagging memories and concepts, machines will simply revert to the simpler concepts we use them for in our recent times, such as stealing cryptocurrency. After all, we live in a world run by capitalism, so it’s only logical. ¯\(ツ)/¯
As a technologist, I have to remind everyone that AI is not intelligence. It’s a word prediction/statistical machine. It’s guessing at a surprisingly good rate what words follow the words before it.
It’s math. All the way down.
We as humans have simply taken these words and have said that it is “intelligence”.
As another technologist, I have to remind everyone that unless you subscribe to some rather fringe theories, humans are also based on standard physics.
Which is math. All the way down.
I agree, the maths argument is not a good one. While a neural network is perhaps closer to what a brain is than just a CPU (or a clock, as it was compared to in he olden days), it would be a very big mistake to equate the two.
As a mathematician, it should be noted that the mathematics of physics aren’t laws of the universe, they are models of the laws of the universe. They’re useful for understanding and predicting, but are purely descriptive, not prescriptive. And as they say, all models are wrong, but some are useful
As a random person on the Internet I don’t actually have anything to add but felt it would be nice to jump in.
That’s true, but that doesn’t contradict the above comment. Unless you believe in something like a spirit or soul, you must concede that human intelligence ultimately arises from physical matter (whatever your model of physics is). From what we know of science right now, there are no direct reasons for thinking that true intelligence or even consciousness is limited to biological organisms based on carbon and could not arise in silicon.
My point was more so that the argument that humans can be modeled with math & physics implies that LLMs are/could become intelligent, conscious things, since they’re also based on math, is nonsense. These are statistical prediction algorithms; they work nothing like a nervous system or a conscious living being. They can be impressive in narrow use cases, like all ML, but they cannot actually learn or perform novel tasks. I don’t think this rules out the possibility of creating some sort of true artificial intelligence, but the current approaches are structurally unable to ever get there, and the conversation above makes really weak points to the contrary. But this was too many words so I figured my other approach was better for brevity lol
Edit: “AI” slop bros stay mad lmao
I generally agree, but I kind of wonder whether something like an advanced LLM has a place as a component of an artificial “brain”. We have a language-focused area in our brain, but we have lots of other components of the brain that does all kinds of other things too. Perhaps we’re “just” missing those other things.
As a philosopher, I have to remind you that humans invented math and physics to model reality.
Humans are not based on physics or math. That would be like saying the earth is based on a globe.
Few of countless dictionary definitions for intelligence:
There isn’t even concensus on what intelligence actually means yet here you are declaring “AI is not intelligence” what ever that even means.
Artificial Intelligence is a term in computer science that describes a system that’s able to perform any task that would normally require human intelligence. Atari chess engine is an intelligent system. It’s narrowly intelligent as opposed to humans that are generally intelligent but it’s intelligent nevertheless.
You’re more precisely right, but also the aforementioned person is not wrong. Intelligence is a broad term as we’re discovering. Truth is, we don’t have the language to effectively communicate about AGI in the ways we’d like to. We don’t know if consciousness is a prerequisite to truly generalizable intelligence, we don’t even know what consciousness is, we don’t know what dimensions truly matter here. Is intelligence a dimension of consciousness, meaning you can have some intelligence without being conscious? What’s the limit, why? … We need some discovery around the taxonomy/topology of consciousness.
I mean, every one of those definitions do not apply to LLMs.
As someone who knows a thing or two about biology I think LLMs strip away >90% of what makes animals think.
As a therapist, I can tell you the only thing holding LLMs back from true intelligence is having to pee and poop. Peeing and pooping is the foundation of all higher level operations. I poured water on my PC and the LLM I was running said “I think” right before committing suicide
I was arguing against it being an intelligence because it lacked the suffering and past experiences that define intelligence. Without pain and suffering, what are we? Not for it being intelligent.
I think you’re conflating intelligence and consciousness. Pain and suffering requires consciousness but intelligence does not imply pain or suffering or happiness. LLMs are already “intelligent” to a certain degree in some aspects, though not generally intelligent like humans. But there is no reason to believe that you couldn’t have a generally intelligent artificial agent that lacks consciousness and thus can feel no pain or suffering.
It’s something like folks calling a mirror intelligent.
Are you anthromorphizing word suggester into a being experiencing things?
No.
Here is a way of describing what I see as ‘the problem’:
An LLM cannot forget things in its base training data set.
Its permanent memory… is totally permanent.
And this memory has a bunch of wrong ideas, a bunch of nonsensical associations, a bunch of false facts, a bunch of meaningless gibberish.
It has no way of evaluating its own knowledge set for consistency, coherence, and stability.
It literally cannot learn and grow, because it cannot realize why it made mistakes, it cannot discard or ammend in a permanent way, concepts that are incoherent, faulty ways of reasoning (associating) things.
Seriously, ask an LLM a trick question, then tell it it was wrong, explain the correct answer, then ask it to determine why it was wrong.
Then give it another similar category of trick question, but that is specifically different, repeat.
The closer you try to get it toward reworking a fundamental axiom it holds to that is flawed, the closer it gets to responding in totally paradoxical, illogical gibberish, or just stuck in some kind of repetetive loop.
… Learning is as much building new ideas and experiences, as it is reevaluating your old ideas and experiences, and discarding concepts that are wrong or insufficient.
Biological brains have neuroplasticity.
So far, silicon ones do not.
Isn’t general intelligence, or more broadly “consciousness,” a prerequisite to that? How would you make an unconscious machine more conscious merely by making mock scenarios that conscious beings necessarily experience?
That’s getting into phenomenology — why is pain an experience of suffering at all? How would you give it pain and suffering without having already made it AGI? We’re still missing the
<current-form> -> AGIstep.The necessity of which is emergent from our culture and biology, as conscious social beings. We’re still missing a vital step.
What is “loss” and “everyday situations” if not just a way we choose to see the world, again as conscious beings.
How do you give it a “concept” at all?
The AI in its current form has the “memory” in some form, but perhaps not the “nagging.” What should do the “nagging” and what should be the target of the “nagging?” How do you conceptually separate the “memory” and the “nagging” from the “being” that you’re trying to create? Is it all part of the same being, or does it initialize the being?
We’re a long way away from AGI, IMO. The exciting thing to me, though, is I don’t think it’s possible to develop AGI without first understanding what makes N(atural)GI. Depending how far away AGI is, we could be on the cusp of some deeply psychologically revealing shit.
Completely agree with all of this.
Especially the last part.
We don’t even understand our brains, our own minds, we still can’t fully agree on what consciousness or sentience… even… are.
We’re certainly making progress on those fronts… but we are a very, very far distance from the finish line.
That finish line would be like… we solved Psychology, we solved Neuroscience, we have a Grand Unified Theory of Mind, etc.
The major thing AI lacks is continuous parallel “prompting” through a variety of channels including sensory, biofeedback, and introspection / meta-thought about internal state and thinking.
AI currently transforms a given input into an output. However it cannot accept new input in the middle of an output. It can’t evaluate the quality of its own reasoning except though trial and error.
If you had 1000 AIs operating in tandem and fed a continuous stream of prompts in the form of pictures, text, meta-inspection, and perhaps a simulation of biomechanical feedback with the right configuration, I think it might be possible to create a system that is a hell of an approximation of sentience. But it would be slow and I’m not sure the result would be any better than a human — you’d introduce a lot of friction to the “thought” process. And I have to assume the energy cost would be pretty enormous.
In the end it would be a cool experiment to be part of, but I doubt that version would be worth the investment.
It could also be that it lacks the machinery to feel any emotions at all. You don’t (normally) have to train people to be afraid of bears or heights or loneliness or boredom. You also don’t (normally) have to train people to have empathy or compassion.
I argue that our obsession with AI is, itself, a misalignment with our environment; it disproportionately tickles psychological reward centers which evolved under unrecognizably different circumstances.
I guess you don’t have children.
You absolutely do have to train them to be afraid of bears, heights, and every fucking thing you can imagine. You absolutely do have to teach them empathy and compassion. There may be some nugget of instinct, but without reinforcement it might as well not exist.
Hah, okay, you got me there. From my understanding, though, that’s mostly because kids are still figuring out what’s “normal”, so their fear instinct isn’t nearly as strong. I guess I should’ve stuck to the more instinctive sources of fear…
Regardless, that’s not really my point. My point is an LLM doesn’t rely on machinery in the same way that a human brain does. That doesn’t make AI “worse” or “better” overall, but it does make it an awful replacement for other humans.
As a random internet user, I want to remind you, are we sure even if humans are that intelligent to begin with? All those steps you give, are not needed for intelligence.
We keep moving the goal post for what intelligence is, and last I saw we have started to divide intelligence into different categories.
LLMs are just “imitate as closely as possible human responses” for good and for bad. And now we are trying to fix that to be as right as possible, when the flaw is that we as humans are mostly always wrong.