ARC Prize 2025: Solving Puzzles With No Pretraining

The bet: no pretraining, one network per puzzle

Most competitive ARC entries lean on scale — pretrain a big model on synthetic ARC-like data, or prompt a frontier LLM. For ARC Prize 2025 I deliberately took the opposite bet. My solution is in the CompressARC lineage: it uses no pretraining, no external data, and no search. Instead, for every single puzzle it initializes a small neural network from random weights and trains it, at test time, on only that puzzle's own train pairs. Then it uses the fitted network to predict that puzzle's test output. Hence the notebook's name: "ARC-AGI Without Pretraining."

CompressARC (Isaac Liao and Jyo Pari, 2025) frames each puzzle as a minimum-description-length compression problem: a tiny (~76K-parameter) network is fit per task using a VAE-style loss so that gradient descent — not combinatorial program search — does the "reasoning." It's an intriguing idea because it treats generalization as compression: the only thing the model ever sees is the handful of demonstration pairs in front of it. My work here builds directly on that method; the interesting part I want to be honest about is that the hard problem for me wasn't the model, it was throughput.

Why the constraints make this hard

ARC Prize 2025 runs on the harder ARC-AGI-2 benchmark. Submissions execute as a Kaggle notebook, offline (no internet, no external APIs), inside a fixed hardware sandbox, and must finish within a 12-hour wall-clock window. Scoring is pass@2: for each test input you submit exactly two guesses, and you get credit if either is an exact grid match. Winning open-source solutions must also be CC0-licensed to be prize-eligible.

Now put those two things together. CompressARC's original blog setup solved puzzles in series, badly underutilized a single RTX 4070, and — as the notebook markdown candidly admits — "blew past the time budget." A method that fits a fresh network per puzzle is embarrassingly parallel, but only if you actually schedule it that way. So the whole engineering story of my notebook is turning a serial, GPU-starved research prototype into something that saturates all the CPUs and GPUs Kaggle gives you and lands inside 12 hours.

What the notebook actually is

To be precise about the source: the network internals — the CompressARC model, the per-puzzle training loop, grid encoding, and the VAE/MDL loss — live in imported modules (solve_task.py, layers.py, preprocessing.py) that ship in the /kaggle/input/compressarc dataset. My notebook is the orchestration and scheduling layer that sits on top of them. It imports preprocessing, calls solve_task.solve_task(...) once per puzzle, and is responsible for memory probing, parallelization, timing, and writing submission.json. One concrete model-side change I made was rewriting layers.direction_share() to run faster, which bought roughly a 5–10% speedup.

The per-puzzle call signature the scheduler drives is essentially:

args = (task_name, "test", end_time, n_iterations,
        gpu_id, memory_dict, solutions_dict, error_queue)
p = multiprocessing.Process(target=solve_task.solve_task, args=args)
p.start()

Each spawned process trains one puzzle's network for n_iterations gradient steps on a chosen gpu_id, and writes its predicted grids back into a shared solutions_dict — which becomes the pass@2 submission. Global setup pins torch.float32, enables cudnn.benchmark and TF32 matmuls, and uses the spawn start method so each CUDA process is clean.

The scheduling strategy

The plan described in the notebook is a three-phase probe-then-pack:

Memory probe (2 steps). Run just 2 optimization steps of every puzzle to measure how much GPU memory each one actually consumes. Puzzles vary in grid size, so their footprints differ.
Timing probe (10 steps). Run 10 steps at the chosen puzzle-parallelization to measure real seconds-per-step under contention.
Main run (as many steps as fit). With per-step cost known and time remaining known, compute the largest step budget that still lands inside 12 hours, then run that. For a 120-puzzle set, the notebook estimates this yields roughly ~2300 optimization steps per puzzle.

The scheduler itself (parallelize_runs) reads each GPU's free memory with torch.cuda.mem_get_info, subtracts a 6 GB safety buffer, and then greedily launches puzzle processes while two constraints hold: the GPU still has enough memory quota for that puzzle's measured usage, and the number of live processes stays under the CPU count. When a process finishes it returns its memory quota to the pool so the next puzzle can start. There's a hard timeout guard (MAX_TASK_RUNTIME = 600s) that terminates any puzzle that runs away, plus a global end-time check so the run stops cleanly before the deadline rather than being killed mid-write.

flowchart TD
  A["Load ARC-AGI-2 test tasks"] --> B["Read free GPU memory, minus 6GB buffer"]
  B --> C["2-step memory probe per puzzle"]
  C --> D["10-step timing probe at chosen parallelization"]
  D --> E["Compute max steps that fit 12h budget"]
  E --> F["Pack puzzles across GPUs under memory + CPU limits"]
  F --> G["Per puzzle: train tiny net from scratch, ~2300 steps"]
  G --> H["Form 2 predictions per test input"]
  H --> I["Write submission.json, pass@2"]

Augmentation and pass@2 formation

CompressARC's fitting relies on equivariance and augmentation of the puzzle's own examples (transposes, color/geometry symmetries) so that a network fit to a few pairs isn't just memorizing pixels. The two required pass@2 attempts per test input are produced by solve_task from the fitted per-puzzle model and collected into solutions_dict; my orchestration layer's job is to make sure every task actually gets its two attempts written before the clock runs out. I want to be transparent that these prediction/augmentation details are implemented inside the imported solve_task module rather than in the notebook cells I authored, so I'm characterizing them at the method level rather than quoting exact transforms.

Honest results and limitations

A few things I won't dress up:

No score is persisted in the notebook. The notebook writes a submission.json but does not record or print a leaderboard score, so I'm not going to quote a number I can't verify from the source. For calibration only: the upstream CompressARC method reports roughly 20% on ARC-AGI-1's eval set and around 4% on the much harder ARC-AGI-2 — ARC-AGI-2 is brutal for no-pretraining methods.
The active cell is more conservative than the ideal. The markdown describes the full probe-then-pack plan targeting ~2300 steps, but the final executable cell in the notebook actually runs a simplified path — a 5-step estimation pass followed by a 20-step main run with a flat 1 GB-per-task memory assumption. That's a safe, deadline-robust fallback, and it's a lower step budget than the aspirational plan. It reflects the real tension: more steps per puzzle means better fits but more deadline risk.
Throughput is the ceiling, not cleverness. Because quality scales with steps-per-puzzle, and steps-per-puzzle is capped by memory-limited parallelism inside 12 hours, the entire competition comes down to how tightly you can pack GPUs — which is exactly why the memory and timing probes exist.

What I like about this entry is that it's an honest, from-scratch bet: no giant pretrained prior, just gradient descent compressing each puzzle on its own terms. It's not the highest-scoring recipe on ARC-AGI-2, but it's a clean demonstration that "learning at test time, per instance" is a real and schedulable strategy — and that on constrained hardware, the systems engineering around a method matters as much as the method itself.

← Back to all posts