Understanding the data and the benchmark

Post-training   · 6 min read

I want to understand post-training properly. Not just which library call to make, but why each method works, down to the maths. So I am going to take one small open model, Qwen2.5-Coder-7B, and teach it to turn English questions into SQL, one technique at a time, writing up each step here.

This first post is the plan and the data. No model runs yet.

The goal

Two things.

Learn each method in depth. Fine-tuning, distillation, DPO and GRPO: derive each one, implement it, and see what it actually does to the model.

See how far reinforcement learning can take a small model. There is already a good bar. Snowflake trained Arctic-Text2SQL-R1-7B with GRPO, rewarding only whether the SQL runs and returns the right rows, and it scores 68.9% on BIRD dev and 68.5% on the hidden test set. It started from a model already tuned for SQL. I am starting from plain Qwen, so I also get to see which step contributes what.

The road ahead

StagePartsWhat I will try
Baselines2 to 4The model as it is, better prompts, and frontier models for comparison
Supervised learning5 to 7Fine-tuning, distilling from a bigger model, voting over samples
Preferences and RL8 and 9DPO, then GRPO
The final exam10BIRD’s hidden test set

The benchmark: BIRD

BIRD is 12,751 questions over 95 real databases, each with the SQL that answers it.

SplitQuestionsWhat it is for
Train9,428Training
Dev1,534Measuring progress, over 11 databases
Test1,789The final score. Hidden: BIRD runs your model for you

I will measure everything on dev until the final part. Its databases range from California schools to Formula 1 to a Czech bank, and its questions are tagged simple (925), moderate (464) or challenging (145).

BIRD also released a cleaned dev set in November 2025, dev-1106, which fixes 451 of the answer keys. I will report both: the original because papers use it, and dev-1106 because it is closer to correct.

All 11 dev databases
DatabaseQuestionsTables
card_games1917
codebase_community1868
formula_117414
thrombosis_prediction1633
student_club1588
toxicology1454
superhero12910
european_football_21298
financial1068
california_schools893
debit_card_specializing646

What one question looks like

The model gets the database’s tables and columns, the question, and usually a short evidence hint. It writes one SQL query, the query runs, and its rows are compared with the answer key’s.

  1. Prompt Tables and columns, the question, the evidence hint
  2. Model Writes one SQL query
  3. Database Runs the query
  4. Answer key Its rows are compared with the right ones

Here is question 5, from the California schools database:

Question: How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual?

Evidence: Exclusively virtual refers to Virtual = 'F'

SELECT COUNT(DISTINCT T2.School)
FROM satscores AS T1
INNER JOIN schools AS T2 ON T1.cds = T2.CDSCode
WHERE T2.Virtual = 'F' AND T1.AvgScrMath > 400

The answer is 4:

SchoolCountyVirtualAverage SAT Math
Capistrano Connections AcademyOrangeF513
California Virtual Academy @ Los AngelesLos AngelesF502
California Connections Academy @ RiponSan JoaquinF487
Insight @ Los AngelesLos AngelesF437

That hint matters more than it looks. 973 schools have an SAT Math average above 400, and the Virtual column holds F, N, P or nothing. Nothing in the schema says F means exclusively virtual. Part 2 measures how much the model loses without it.

Three difficulties

Harder questions are not about harder topics. The SQL just has to do more. Three examples, exactly as BIRD writes them:

DifficultyQuestion and evidenceAnswer-key SQLAnswer
SimpleHow many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? Exclusively virtual refers to Virtual = ‘F’SELECT COUNT(DISTINCT T2.School) FROM satscores AS T1 INNER JOIN schools AS T2 ON T1.cds = T2.CDSCode WHERE T2.Virtual = 'F' AND T1.AvgScrMath > 4004
ModerateHow many accounts who choose issuance after transaction are staying in East Bohemia region? A3 contains the data of region; ‘POPLATEK PO OBRATU’ represents for ‘issuance after transaction’.SELECT COUNT(T2.account_id) FROM district AS T1 INNER JOIN account AS T2 ON T1.district_id = T2.district_id WHERE T1.A3 = 'east Bohemia' AND T2.frequency = 'POPLATEK PO OBRATU'13
ChallengingFor the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male clients? Percentage of the male clients = DIVIDE(COUNT(male clients), COUNT(clients)) * 100; Male refers to gender = ‘M’, A3 is the region name. A4 contains the information about inhabitants.SELECT CAST(SUM(T1.gender = 'M') AS REAL) * 100 / COUNT(T1.client_id) FROM client AS T1 INNER JOIN district AS T2 ON T1.district_id = T2.district_id WHERE T2.A3 = 'south Bohemia' GROUP BY T2.A4 ORDER BY T2.A4 DESC LIMIT 144.26

In the moderate one the data is in Czech, and only the hint says 'POPLATEK PO OBRATU' means issuance after transaction. The challenging one is also an answer key dev-1106 corrects: it sorts population as text and picks the wrong district.

How the model is run

For now, every run is a single shot:

TurnsOne prompt in, one answer out
AttemptsOne, with greedy decoding, so the same prompt always gets the same answer
ToolsNone. The model cannot run a query or look at the data
FeedbackNone. It never sees whether its SQL ran

That is the hard way to play, on purpose: any improvement has to come from the model or the prompt. A lot of the leaderboard’s top systems are agents instead, which look at the data, run a draft and fix their errors. I want to try that too, later, and see how it compares with what training buys.

What comes next

Part 2 runs Qwen2.5-Coder-7B with no training, both Base and Instruct, with and without the evidence hint. It also covers how answers are scored, and why every score is a few points of noise either way.

BIRD is by Jinyang Li and colleagues (arXiv 2305.03111) and is shared under CC BY-SA 4.0. The questions, SQL and rows quoted here are from it.

← All writing