Blog · Kaggle
Beating 24% on ARC-AGI-2: A Hybrid Symbolic + Neural Solver, Honestly
The honest starting point
ARC-AGI-2 is an open research frontier, and I want to be candid about that from the first sentence. As of the 2025 competition, the best self-contained Kaggle solution scored 24.03% — that was NVARC, from the NVIDIA Kaggle Grandmasters, using a Qwen3-4B model with a 16-token vocabulary, roughly 100k synthetic tasks, and test-time training. Average human performance on the public eval is about 66%. The grand prize ($1M) needs 85%, and nobody is close.
So "beating the top solution" means beating ~24% inside a brutal set of constraints, not building something that solves ARC. The rules for the ARC-AGI-2 track:
- A Kaggle notebook, under 12 hours of runtime, no internet, and no API models — the model must be fully self-contained and offline (no GPT/Claude API).
- Open source under CC0/MIT-0 to be eligible.
- The metric is pass@2 per test input, and a task counts only if every test input is solved. (45 of the 120 public-eval tasks have multiple test inputs — a subtle multiplier on difficulty.)
This post is about the architecture I built toward that target, and — just as importantly — about what I could and couldn't verify without a GPU.
Two facts that decided the whole design
Before writing a line of the neural pipeline, I established two things locally, and both turned out to be load-bearing.
Fact 1: symbolic search cannot compete on ARC-AGI-2. I built a battery of 19 inducer families — colormaps, dihedral transforms, scaling, tiling, symmetry repair, boolean panel ops, object selection, fractals, gravity, dedup, plus a compositional search that chains parameter-free ops. Every inducer only ever emits a rule that reproduces all training pairs exactly, so precision is essentially 100%. On the 1000 training tasks it solves about 6%. On the 120 public-eval tasks it solves 0. Not a rounding-to-zero — literally none. No eval task admits a single verified symbolic rule. ARC-AGI-2 was explicitly built to defeat this class of solver, and my own numbers reproduce that wall exactly. The neural model has to be the primary scorer.
Fact 2: the bottleneck is selection, not generation. The TRM paper reports pass@1 ≈ 18% but pass@128 ≈ 30%. In other words, the model often produces the correct grid but fails to rank it in the top 2. A strong selection layer is worth several points essentially for free.
Those two facts — symbolic scores zero, and generation beats selection — are the entire justification for a hybrid.
The architecture: three engines, one selector
The system is a hybrid of three engines feeding a single selector that outputs the final two attempts.
flowchart TD T["Task: train pairs + test input"] --> A["Engine A — Symbolic DSL battery (19 families, CPU, verified rules)"] T --> B["Engine B — Neural TTT model (Qwen-4B, 16-token vocab, LoRA per task)"] T --> C["Engine C — Test-time augmentation (8 dihedral × color perms)"] C --> A C --> B A --> D["Engine D — Selector (verify + consensus + likelihood)"] B --> D D --> E["Final: pass@2 attempts"]
- Engine A — symbolic battery (
arc/solvers.py): only emits hypotheses that reproduce all train pairs, giving ~100% precision. It rarely fires on eval, but each hit is a near-certain pass@2 slot at ~zero GPU cost. - Engine B — neural test-time training (
kaggle/arc_agi2_pipeline.py): the primary scorer. Per task, it LoRA-fine-tunes a 16-token causal LM on the task's own augmented train pairs, then samples. - Engine C — augmentation (
arc/augment.py): the 8 dihedral symmetries times N color permutations. It's shared — it expands the TTT data and creates diverse inference frames for both A and B. - Engine D — selector (
arc/ensemble.py): fuses symbolic verification, augmentation consensus, and model likelihood into the final two attempts.
Everything CPU-checkable is unit-tested locally. The GPU engine lives in kaggle/ and is designed to run inside a Kaggle kernel — offline, under 12 hours.
Edge 1: DSL-grounded synthetic data with verified difficulty control
NVARC built ~100k tasks by LLM-combining ~800 seed task descriptions, then filtering by code-agreement (≥8/20 implementations must agree) — expensive and noisy. I went the other way. I run my forward DSL primitives generatively (arc/generate.py): sample a program (a pipeline of k primitives with fixed parameters), sample k random input grids, and apply the program to each. Every task is therefore verified-by-construction — the rule is identical across all pairs by definition. Measured cost: 0.5 ms/task (100k tasks in under a minute, millions in minutes), with zero invalid tasks.
The real prize is difficulty control with empirical validation. I pointed my own symbolic solver at the generated tasks and measured how often it recovers the true rule as a function of program depth:
| Program depth | Symbolic recovery |
|---|---|
| 1 | 81% |
| 2 | 14% |
| 3 | 3% |
| 4 | 1% |
This is the mechanism in miniature. Compositional depth is exactly what defeats pattern-matching — the same reason eval sits at 0% for symbolic search. So I skew the training curriculum toward depth 3–4 to force the model to learn composition, the ARC-AGI-2 differentiator, rather than memorizing single transforms. And because the generator is mine, I can also match eval's hard surface statistics directly: large inputs to small outputs, many colors, legend/key indirection.
Edge 2: selection-first inference
This edge attacks the pass@1 ≪ pass@k gap head-on. Instead of trusting one greedy decode, I generate many candidates — sampling across the 8 dihedral frames times color permutations — de-augment each back into the original frame, and then pick the final two by augmentation consensus: the grid the model produces consistently across frames is far more likely correct than any single decode. The logic lives in arc.ensemble.consensus_rank, and it's unit-tested.
Conceptually, this converts the documented pass@128 ≈ 30% generation ceiling into realized pass@2 — points NVARC left on the table by prioritizing a single model decode under the time budget. If the model already can produce the answer, consensus is how you surface it.
Edge 3: the hybrid verified ensemble
Here's where the three engines pay their rent together. When the symbolic battery finds a rule that fits all train pairs, it takes attempt slot 1 — a near-certain point at ~zero GPU cost — and the neural consensus fills slot 2. I verified locally that a symbolic rule rescues a task even when a (mock) neural model votes mostly wrong (arc.ensemble.select_pass2). The selection priority is: symbolic-verified answer first, neural consensus second, remaining symbolic candidates as backfill. It's a strict upgrade: symbolic can only ever help, never hurt.
An earlier attempt worth naming
My 2025 Kaggle kernel (improved-arc-agi-without-pretraining) was a genuinely different beast: a per-puzzle, no-pretraining approach in the CompressARC lineage. The engineering focus there was pure throughput — profiling each puzzle's memory in a 2-step probe, finding the optimal puzzle-parallelization under the memory ceiling, then packing as many optimization steps as possible into 12 hours (roughly 2300 steps per puzzle on 120 tasks) across all available CPUs and GPUs. It taught me the budget discipline that shows up in the 2026 pipeline's time-budget guard, but it's architecturally distinct — no synthetic pretraining, no symbolic layer, no cross-frame consensus. The 2026 hybrid is a clean break toward the NVARC-class recipe plus my three edges.
Honest status and risks
I'll end where I started — candidly.
- Engines A + C alone, no GPU: ~0% on eval. The local symbolic-only submission is a placeholder-grid baseline, and that's the whole point of ARC-AGI-2. The symbolic layer is valuable only as the precision slot in the ensemble.
- Engine B (an NVARC-class reproduction): the realistic floor is "approach ~24%" once the GPU pipeline runs. Beating it depends on Edges 1–2 actually adding points, which can only be confirmed by Kaggle GPU runs — not on my Mac.
- Biggest risk: the selection gains are empirical. If neural candidates are low-diversity, consensus adds little. Mitigation: tune sampling temperature and augmentation count to maximize candidate diversity before selection.
- Compute risk: test-time training a 4B model within 12 hours for 120 tasks is tight. LoRA, small step counts, and batching are mandatory, and the pipeline falls back to symbolic-only if it runs out of budget.
The verified scaffolding is done: the grid library, the exact pass@2 scorer, the 19-family battery, the augmentation round-trips, the 0.5 ms/task generator with difficulty control, and the unit-tested selector. What remains is the honest part I can't fake — pretraining the 16-token model and getting the first real neural number from inside Kaggle. ARC-AGI-2 doesn't hand out points for a good architecture diagram. But if I'm going to chase a genuine research result, I'd rather build it on measurements than on hope.
← Back to all posts