The starting line: a 7B against the frontier

Post-training   · 9 min read

Before training anything, I need to know where the model starts. So this part runs Qwen2.5-Coder-7B exactly as it comes, on all 1,534 BIRD dev questions, next to three much bigger models given the same prompts. Every model answers each question twice: once without BIRD’s evidence hint, once with it.

The five models

  • Qwen2.5-Coder-7B, the base model. It is not a chat model, so it gets the prompt as plain text ending in an opened sql code block, and carries on from there.
  • Qwen2.5-Coder-7B-Instruct, the same model after Qwen’s own instruction tuning. It gets the prompt as a chat message.
  • Three frontier models for comparison: Beam 1.0 (501B parameters, 23B used per token), DeepSeek V4.1-Flash, and Nemotron 3.5 Super (120B, 12B used per token). They get the same chat message as Instruct. DeepSeek thinks out loud before answering, so it gets up to 65,536 tokens per answer instead of 16,384, enough that it rarely runs out before writing the SQL.

All five get one try per question, with the randomness turned off. This is the whole prompt for question 5 from Part 1, with the schema shortened:

You are given a SQLite database and a question about it. Write one SQL query that answers the question.

Database schema:
CREATE TABLE satscores (
  cds TEXT,
  AvgScrMath INTEGER,
  ...
);

CREATE TABLE schools (
  CDSCode TEXT,
  Virtual TEXT,
  ...
);

Evidence: Exclusively virtual refers to Virtual = 'F'

Question: How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual?

Return only the SQL query, in a ```sql code block.

Without the hint, the Evidence line is left out. Nothing else changes.

How an answer is scored

The last SQL block in the model’s answer is run on the real database, and its rows are compared with the rows the answer key returns. It is correct if the two sets of rows are equal. Order and duplicates do not count, the same rule as BIRD’s own scoring script. Everything else is wrong, and gets one label saying how:

Wrong resultThe SQL ran but returned different rows
SQL errorSQLite rejected the query
TimeoutIt ran for more than 30 seconds
No SQLThere was no SQL block in the answer
TruncatedThe answer hit the length limit before the SQL

The score is the share of the 1,534 questions marked correct.

The results

ModelWithout the hintWith the hintWhat the hint adds
Beam 1.050.0%63.6%+13.6
DeepSeek V4.1-Flash46.9%62.2%+15.3
Nemotron 3.5 Super36.0%56.1%+20.0
Qwen2.5-Coder-7B-Instruct32.4%49.8%+17.4
Qwen2.5-Coder-7B26.7%38.7%+12.0

Three things stand out.

The hint is worth more than most of the size gap. It adds 12 to 20 points to every model. With it, the 7B scores 49.8%, level with Beam’s 50.0% without it.

Instruction tuning is worth 11 points. With the hint, Instruct beats Base by 11 points, so every later part starts from Instruct.

There are 14 points to close. With the hint, Instruct is 13.8 points behind Beam. That is the gap training has to close, and Arctic-Text2SQL-R1’s 68.9% is 5 points past that again.

Without the hint · accuracy with its 95% Wilson interval
Beam 1.0 50.00% ±2.50
DeepSeek V4.1-Flash 46.94% ±2.49
Nemotron 3.5 Super 36.05% ±2.40
Qwen 7B Instruct 32.40% ±2.34
Qwen 7B Base 26.66% ±2.21
Data
RunCorrectAccuracy95% interval
Beam 1.0767 / 1,53450.00%47.50% to 52.50%
DeepSeek V4.1-Flash720 / 1,53446.94%44.45% to 49.44%
Nemotron 3.5 Super553 / 1,53436.05%33.68% to 38.48%
Qwen 7B Instruct497 / 1,53432.40%30.10% to 34.78%
Qwen 7B Base409 / 1,53426.66%24.51% to 28.93%
With the hint · accuracy with its 95% Wilson interval
Beam 1.0 63.56% ±2.41
DeepSeek V4.1-Flash 62.19% ±2.42
Nemotron 3.5 Super 56.06% ±2.48
Qwen 7B Instruct 49.80% ±2.50
Qwen 7B Base 38.66% ±2.43
Data
RunCorrectAccuracy95% interval
Beam 1.0975 / 1,53463.56%61.12% to 65.93%
DeepSeek V4.1-Flash954 / 1,53462.19%59.74% to 64.58%
Nemotron 3.5 Super860 / 1,53456.06%53.57% to 58.53%
Qwen 7B Instruct764 / 1,53449.80%47.31% to 52.30%
Qwen 7B Base593 / 1,53438.66%36.25% to 41.12%

BIRD dev, 1,534 questions. Both charts use the same scale.

How sure are these numbers?

Start with a coin. Flip it 10 times and get 6 heads, and nobody would say it lands heads 60% of the time. Flip it 1,534 times and the share of heads gets close to the coin’s real tendency, but never exactly.

A benchmark works the same way. Each question is one flip: the model gets it right or wrong. Instruct got 764 of 1,534 right, 49.8%. On 1,534 different questions of the same kind, it would have scored a point or two higher or lower. A 95% interval is the range the model’s real skill very likely falls in, here 47.3% to 52.3%. The 95% describes the method: run the experiment on 100 fresh sets of questions, and about 95 of the 100 ranges would contain the real value. The whiskers on the charts above are these intervals.

Where the ±2.5 comes from. With nn flips and a chance pp of heads, the share of heads typically wobbles by p(1−p)/n\sqrt{p(1-p)/n}. For p=0.5p = 0.5 and n=1,534n = 1{,}534 that is 1.28 points. About 95% of results land within 1.96 wobbles of the truth, so with p^\hat p the score:

p^±1.96p^(1−p^)n=49.8%±2.5\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}} = 49.8\% \pm 2.5

Why not stop there? That formula breaks with few questions, or with scores near 0% or 100%:

ResultThat formulaWilson
1 right out of 10−8.6% to 28.6%1.8% to 40.4%
0 right out of 100% to 0%0% to 27.8%
764 right out of 1,53447.3% to 52.3%47.3% to 52.3%

A score below zero is impossible, and 0 out of 10 does not prove a model can never get one right. The Wilson interval fixes both and gives the same answer in the ordinary case, so it is what every interval in this series uses. With z=1.96z = 1.96 for 95%:

p^+z22n±zp^(1−p^)n+z24n21+z2n\frac{\hat p + \dfrac{z^2}{2n} \pm z\sqrt{\dfrac{\hat p(1-\hat p)}{n} + \dfrac{z^2}{4n^2}}}{1 + \dfrac{z^2}{n}}

What it means for the results. Every score here is good to about ±2.5 points. The 7B with the hint and Beam without it are 0.2 points apart, well inside that, so they are level. The 14 points between Instruct and Beam with the hint are far outside it. Try other scores:

Try it · the 95% interval for any score

49.80%, with a 95% interval of 47.31% to 52.30%: ±2.50 points.

When the hint makes it worse

The hint’s gain is a net number. Compare each question with and without it, and the hint fixes a lot of questions but also breaks some, for every model:

ModelHint fixedHint broke
Beam 1.029082
DeepSeek V4.1-Flash29258
Nemotron 3.5 Super34841
Qwen2.5-Coder-7B-Instruct31346
Qwen2.5-Coder-7B25167

“Broke” means right without the hint and wrong with it. For Instruct there are 46 of those, and I read all 46:

What happenedQuestions
The hint’s value does not match the data (case, spelling, date format), and the model copied it6
The hint uses notation SQLite cannot run, or a formula that leads to broken SQL5
The hint contradicts the answer key4
The hint says max(...) or min(...) and the model used the aggregate the wrong way3
The hint names extra columns and the model returned them too2
The question has no hint, so the prompt was identical, yet the answer changed (run-to-run noise on the GPU)1
No clear link: the longer prompt sent the model to a different table or join25

In 20 of the 46, the hint itself is to blame. Four of them, with both answers:

Right without the hint, wrong with it 4 records
916 wrong case The hint says 'italian'. The database stores 'Italian'.
Question
Please list the surnames of all the Italian drivers.
Hint
Italian refers to nationality = 'italian'
Without the hint
SELECT surname FROM drivers WHERE nationality = 'Italian';
Result
97 rows. Correct.
With the hint
SELECT surname FROM drivers WHERE nationality = 'italian';
Result
No rows. SQLite compares text exactly, capital letters included, so nothing matches.
677 not SQLite The hint uses YEAR(), which SQLite does not have.
Question
How many posts did Jay Stevens have in 2010?
Hint
DisplayName = 'Jay Stevens'; in 2010 refers to YEAR(CreationDate) = 2010;
Without the hint
SELECT COUNT(*)
FROM posts
JOIN users ON posts.OwnerUserId = users.Id
WHERE users.DisplayName = 'Jay Stevens'
  AND STRFTIME('%Y', posts.CreaionDate) = '2010';
Result
3. Correct.
With the hint
SELECT COUNT(*)
FROM posts
JOIN users ON posts.OwnerUserId = users.Id
WHERE users.DisplayName = 'Jay Stevens' AND YEAR(posts.CreaionDate) = 2010;
Result
Error: no such function.
1490 integer division Following the hint's formula in order rounds the answer down to 0.
Question
How many percent of LAM customer consumed more than 46.73?
Hint
Percentage of LAM customer consumed more than 46.73 = (Total no. of LAM customers who consumed more than 46.73 / Total no. of LAM customers) * 100.
Without the hint
SELECT CAST(SUM(CASE WHEN T2.Consumption > 46.73 THEN 1 ELSE 0 END) AS REAL) * 100 / COUNT(T1.CustomerID) AS Percentage
FROM customers AS T1
JOIN yearmonth AS T2 ON T1.CustomerID = T2.CustomerID
WHERE T1.Segment = 'LAM';
Result
98.53. Correct.
With the hint
SELECT (SUM(CASE WHEN T2.Consumption > 46.73 THEN 1 ELSE 0 END) / COUNT(T1.CustomerID)) * 100 AS Percentage
FROM customers AS T1
JOIN yearmonth AS T2 ON T1.CustomerID = T2.CustomerID
WHERE T1.Segment = 'LAM';
Result
0. Dividing one whole number by another, SQLite drops the fraction before the * 100.
788 contradicts the key The hint's formula and the answer key compute different percentages.
Question
How many percent of female heroes were published by Marvel Comics?
Hint
percent = MULTIPLY(DIVIDE(SUM(gender = 'Female' WHERE publisher_name = 'Marvel Comics'), COUNT(publisher_name = 'Marvel Comics')), 100); ...
Without the hint
SELECT CAST(SUM(CASE WHEN T2.publisher_name = 'Marvel Comics' THEN 1 ELSE 0 END) AS REAL) * 100 / COUNT(*) AS percentage
FROM superhero AS T1
JOIN publisher AS T2 ON T1.publisher_id = T2.id
JOIN gender AS T3 ON T1.gender_id = T3.id
WHERE T3.gender = 'Female';
Result
54.95: the share of female heroes who are Marvel. Correct.
With the hint
SELECT CAST(SUM(CASE WHEN T1.gender = 'Female' THEN 1 ELSE 0 END) AS REAL) * 100 / COUNT(*) AS percent
FROM gender AS T1
JOIN superhero AS T2 ON T1.id = T2.gender_id
JOIN publisher AS T3 ON T2.publisher_id = T3.id
WHERE T3.publisher_name = 'Marvel Comics';
Result
28.68: the share of Marvel heroes who are female, exactly what the hint asks for. Wrong by the answer key.

What comes next

Part 3 keeps the model fixed and changes the prompt instead: column descriptions, example values, values from the question found in the database, and asking the model to reason before it answers. One at a time, to see what each is worth.

BIRD is by Jinyang Li and colleagues (arXiv 2305.03111) and is shared under CC BY-SA 4.0. The questions, SQL and rows quoted here are from it.

← All writing