Charting Student Math Misunderstandings with DeBERTa

The problem

Kaggle's MAP — Charting Student Math Misunderstandings competition asks a deceptively human question: when a student picks a multiple-choice answer and writes a sentence or two explaining why, can we predict the specific misconception behind their reasoning? Each training row gives you a math question, the student's selected answer, and their free-text explanation. The label is a combined Category:Misconception string — for example True_Correct:NA or a false category paired with a named misconception.

The scoring metric is MAP@3 (mean average precision at 3). You submit up to three labels per row, and you get full credit if the true label is your first guess, half credit if it's second, and a third of the credit if it's third. That structure completely shapes the solution: it's a ranking problem, not a single-label classification problem, and the model that spreads probability mass sensibly across its top few guesses wins.

When I started, the top public notebooks leaned on TF-IDF. My bet was that an encoder LLM — DeBERTa-v3 — would earn a better CV score by actually reading the explanation. That bet paid off, and the resulting starter notebook (Deberta Starter - LB 0.925 | CV 0.930+) picked up 105 upvotes from the community.

Turning the task into text

The core idea is to reframe misconception prediction as sequence classification. I collapse the two label columns into one target and encode it with a LabelEncoder:

train.Misconception = train.Misconception.fillna('NA')
train['target'] = train.Category + ":" + train.Misconception
train['label']  = LabelEncoder().fit_transform(train['target'])

That single line — Category + ":" + Misconception — is the whole label design. Missing misconceptions become the literal token NA, so a correct answer with no misconception is just another class the model can predict.

The model's input is a plain formatted string. In the flagship starter I keep it simple:

def format_input_v2(row):
    return (
        f"Question: {row['QuestionText']}\n"
        f"Answer: {row['MC_Answer']}\n"
        f"Student Explanation: {row['StudentExplanation']}"
    )

One design decision I made deliberately: I removed any explicit "this answer is correct/incorrect" sentence from the training text. I wanted the model to infer correctness from context rather than be spoon-fed it. I still compute a correctness flag though — by finding, per QuestionId, the most frequent answer that appears under a True_* category — because later notebook variants fold it back into the prompt.

flowchart TD
  A[QuestionText] --> D[format_input_v2 string]
  B[MC_Answer + options] --> D
  C[StudentExplanation] --> D
  D --> E[DeBERTa-v3 tokenizer, MAX_LEN 256]
  E --> F[DebertaV2ForSequenceClassification, n_classes]
  F --> G[Softmax over Category-Misconception classes]
  G --> H[argsort top-3]
  H --> I[Top-3 labels joined for submission]

The model and training

Everything runs through Hugging Face's DebertaV2ForSequenceClassification with a classification head sized to the number of Category:Misconception classes. Tokenization caps at MAX_LEN = 256, which comfortably covers almost every row — a quick token-length histogram showed only a handful of samples getting truncated.

The starter's headline finding is about model size versus score:

DeBERTa-v3-xsmall → CV 0.920+ (I run this for 10 epochs)
DeBERTa-v3-base → CV 0.930+ (6 epochs is the sweet spot)

Key training choices: learning rate 5e-5, cosine LR schedule with a 10% warmup ratio, dynamic padding via DataCollatorWithPadding (no wasteful max_length padding), and — crucially — metric_for_best_model="map@3" with load_best_model_at_end=True. I wrote a custom compute_map3 so the Trainer selects checkpoints on the exact competition metric rather than accuracy:

top3 = np.argsort(-probs, axis=1)[:, :3]
match = (top3 == labels[:, None])
# 1.0 if rank-1, 0.5 if rank-2, 1/3 if rank-3

At inference, the flow mirrors training exactly: same format_input_v2, softmax the logits, take the top-3 class indices with argsort, inverse-transform back to label strings, and join them into the Category:Misconception submission column.

Scaling up: large models and richer prompts

From the starter I pushed in two directions across separate notebooks.

Richer input. In the question-options variant I stopped showing only the student's chosen answer and instead injected all the multiple-choice options for that question, plus an explicit correctness flag:

Question: ...
Options: (A) ... (B) ... (C) ... (D) ...
Student Selected Answer: ...
Correct? Yes/No
Student Explanation: ...

The intuition: knowing the full option set and whether the student got it right gives the model far more signal about which misconception is plausible.

Bigger backbone. Moving to DeBERTa-v3-large meant fighting memory. On a single GPU I dropped per-device batch size to 2 with gradient_accumulation_steps=8, turned on fp16 and gradient_checkpointing, and trained 3 epochs. On a 4×L4 setup I could push batch size to 64 (effective 256), switch to bf16, and bring a full training run down to roughly 4 hours. One large fine-tune landed around CV 0.926.

I also built a dedicated preprocessing and features notebook — math-aware text cleaning (fraction tokenization like 1/2 → FRAC_1_2), lemmatization, numeric length/ratio features, and a TF-IDF matrix — saved as reusable artifacts. It's the classic-ML complement to the transformer track and useful for anyone wanting to blend a lightweight model in.

Ensembling

The final piece is an inference-only ensemble notebook that loads four fine-tuned checkpoints and blends their probabilities:

Model	Reported score
DeBERTa-v3-xsmall (run 1)	LB 0.925
DeBERTa-v3-xsmall (run 2)	LB 0.931
DeBERTa-v3-large	LB 0.927
DeBERTa-v3-base	0.920

I implemented ten blending strategies — simple average, performance-weighted, performance-squared weighting, rank averaging, power averaging, temperature scaling, geometric mean, confidence weighting, best-model-only, and a top-2 blend. The default I ship with is performance-squared weighting: it leans on the stronger models while still capturing ensemble diversity. Because the leaderboard enforces a ~9-hour, ~16,000-sample inference budget, the notebook explicitly times itself and confirms it stays inside the limit.

flowchart LR
  M1[xsmall run 1 probs] --> W[Weighted-squared blend]
  M2[xsmall run 2 probs] --> W
  M3[large probs] --> W
  M4[base probs] --> W
  W --> T[Top-3 argsort]
  T --> S[submission.csv]

Results and honest lessons

The flagship DeBERTa starter reached LB 0.925 with CV 0.930+ and, as a public notebook, earned 105 upvotes — my main evidence that framing this as "let the encoder read the explanation" resonated with the community over the prevailing TF-IDF approaches.

A few things I'd underline honestly:

Optimize for the real metric. Selecting checkpoints on MAP@3 rather than accuracy directly matters when the metric rewards a good top-3, not just a good top-1.
Bigger isn't automatically better. DeBERTa-v3-base at 6 epochs (CV 0.930+) held its own against a far more expensive large model (~CV 0.926); the large model's real value showed up as diversity inside the ensemble.
Split the workflow. The most reliable Kaggle pattern here is train in one notebook, save the model to a dataset, and run a separate internet-off inference notebook — which is exactly how the ensemble stage is structured.
The CV/LB scores above come straight from my notebooks and the checkpoint configs; treat them as the reported numbers for those specific runs rather than a single tuned final submission.

The through-line across all five notebooks is the same simple reframing: concatenate the question, the options, and the student's own words; let DeBERTa rank the misconceptions; take the top three. Everything else — richer prompts, larger backbones, ten ways to blend — is refinement on that one idea.

← Back to all posts