The starting line: a 7B against the frontier
Before training anything, I need to know where the model starts. So this part runs Qwen2.5-Coder-7B exactly as it comes, on all 1,534 BIRD dev questions, next to three much bigger models given the same prompts. Every model answers each question twice: once without BIRD’s evidence hint, once with it.
The five models
- Qwen2.5-Coder-7B, the base model. It is not a chat model, so it gets the prompt as plain text ending in an opened
sqlcode block, and carries on from there. - Qwen2.5-Coder-7B-Instruct, the same model after Qwen’s own instruction tuning. It gets the prompt as a chat message.
- Three frontier models for comparison: Beam 1.0 (501B parameters, 23B used per token), DeepSeek V4.1-Flash, and Nemotron 3.5 Super (120B, 12B used per token). They get the same chat message as Instruct. DeepSeek thinks out loud before answering, so it gets up to 65,536 tokens per answer instead of 16,384, enough that it rarely runs out before writing the SQL.
All five get one try per question, with the randomness turned off. This is the whole prompt for question 5 from Part 1, with the schema shortened:
You are given a SQLite database and a question about it. Write one SQL query that answers the question.
Database schema:
CREATE TABLE satscores (
cds TEXT,
AvgScrMath INTEGER,
...
);
CREATE TABLE schools (
CDSCode TEXT,
Virtual TEXT,
...
);
Evidence: Exclusively virtual refers to Virtual = 'F'
Question: How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual?
Return only the SQL query, in a ```sql code block.
Without the hint, the Evidence line is left out. Nothing else changes.
How an answer is scored
The last SQL block in the model’s answer is run on the real database, and its rows are compared with the rows the answer key returns. It is correct if the two sets of rows are equal. Order and duplicates do not count, the same rule as BIRD’s own scoring script. Everything else is wrong, and gets one label saying how:
| Wrong result | The SQL ran but returned different rows |
|---|---|
| SQL error | SQLite rejected the query |
| Timeout | It ran for more than 30 seconds |
| No SQL | There was no SQL block in the answer |
| Truncated | The answer hit the length limit before the SQL |
The score is the share of the 1,534 questions marked correct.
The results
| Model | Without the hint | With the hint | What the hint adds |
|---|---|---|---|
| Beam 1.0 | 50.0% | 63.6% | +13.6 |
| DeepSeek V4.1-Flash | 46.9% | 62.2% | +15.3 |
| Nemotron 3.5 Super | 36.0% | 56.1% | +20.0 |
| Qwen2.5-Coder-7B-Instruct | 32.4% | 49.8% | +17.4 |
| Qwen2.5-Coder-7B | 26.7% | 38.7% | +12.0 |
Three things stand out.
The hint is worth more than most of the size gap. It adds 12 to 20 points to every model. With it, the 7B scores 49.8%, level with Beam’s 50.0% without it.
Instruction tuning is worth 11 points. With the hint, Instruct beats Base by 11 points, so every later part starts from Instruct.
There are 14 points to close. With the hint, Instruct is 13.8 points behind Beam. That is the gap training has to close, and Arctic-Text2SQL-R1’s 68.9% is 5 points past that again.
Data
| Run | Correct | Accuracy | 95% interval |
|---|---|---|---|
| Beam 1.0 | 767 / 1,534 | 50.00% | 47.50% to 52.50% |
| DeepSeek V4.1-Flash | 720 / 1,534 | 46.94% | 44.45% to 49.44% |
| Nemotron 3.5 Super | 553 / 1,534 | 36.05% | 33.68% to 38.48% |
| Qwen 7B Instruct | 497 / 1,534 | 32.40% | 30.10% to 34.78% |
| Qwen 7B Base | 409 / 1,534 | 26.66% | 24.51% to 28.93% |
Data
| Run | Correct | Accuracy | 95% interval |
|---|---|---|---|
| Beam 1.0 | 975 / 1,534 | 63.56% | 61.12% to 65.93% |
| DeepSeek V4.1-Flash | 954 / 1,534 | 62.19% | 59.74% to 64.58% |
| Nemotron 3.5 Super | 860 / 1,534 | 56.06% | 53.57% to 58.53% |
| Qwen 7B Instruct | 764 / 1,534 | 49.80% | 47.31% to 52.30% |
| Qwen 7B Base | 593 / 1,534 | 38.66% | 36.25% to 41.12% |
BIRD dev, 1,534 questions. Both charts use the same scale.
How sure are these numbers?
Start with a coin. Flip it 10 times and get 6 heads, and nobody would say it lands heads 60% of the time. Flip it 1,534 times and the share of heads gets close to the coin’s real tendency, but never exactly.
A benchmark works the same way. Each question is one flip: the model gets it right or wrong. Instruct got 764 of 1,534 right, 49.8%. On 1,534 different questions of the same kind, it would have scored a point or two higher or lower. A 95% interval is the range the model’s real skill very likely falls in, here 47.3% to 52.3%. The 95% describes the method: run the experiment on 100 fresh sets of questions, and about 95 of the 100 ranges would contain the real value. The whiskers on the charts above are these intervals.
Where the ±2.5 comes from. With flips and a chance of heads, the share of heads typically wobbles by . For and that is 1.28 points. About 95% of results land within 1.96 wobbles of the truth, so with the score:
Why not stop there? That formula breaks with few questions, or with scores near 0% or 100%:
| Result | That formula | Wilson |
|---|---|---|
| 1 right out of 10 | −8.6% to 28.6% | 1.8% to 40.4% |
| 0 right out of 10 | 0% to 0% | 0% to 27.8% |
| 764 right out of 1,534 | 47.3% to 52.3% | 47.3% to 52.3% |
A score below zero is impossible, and 0 out of 10 does not prove a model can never get one right. The Wilson interval fixes both and gives the same answer in the ordinary case, so it is what every interval in this series uses. With for 95%:
What it means for the results. Every score here is good to about ±2.5 points. The 7B with the hint and Beam without it are 0.2 points apart, well inside that, so they are level. The 14 points between Instruct and Beam with the hint are far outside it. Try other scores:
49.80%, with a 95% interval of 47.31% to 52.30%: ±2.50 points.
When the hint makes it worse
The hint’s gain is a net number. Compare each question with and without it, and the hint fixes a lot of questions but also breaks some, for every model:
| Model | Hint fixed | Hint broke |
|---|---|---|
| Beam 1.0 | 290 | 82 |
| DeepSeek V4.1-Flash | 292 | 58 |
| Nemotron 3.5 Super | 348 | 41 |
| Qwen2.5-Coder-7B-Instruct | 313 | 46 |
| Qwen2.5-Coder-7B | 251 | 67 |
“Broke” means right without the hint and wrong with it. For Instruct there are 46 of those, and I read all 46:
| What happened | Questions |
|---|---|
| The hint’s value does not match the data (case, spelling, date format), and the model copied it | 6 |
| The hint uses notation SQLite cannot run, or a formula that leads to broken SQL | 5 |
| The hint contradicts the answer key | 4 |
The hint says max(...) or min(...) and the model used the aggregate the wrong way | 3 |
| The hint names extra columns and the model returned them too | 2 |
| The question has no hint, so the prompt was identical, yet the answer changed (run-to-run noise on the GPU) | 1 |
| No clear link: the longer prompt sent the model to a different table or join | 25 |
In 20 of the 46, the hint itself is to blame. Four of them, with both answers:
916 wrong case The hint says 'italian'. The database stores 'Italian'.
- Question
- Please list the surnames of all the Italian drivers.
- Hint
- Italian refers to nationality = 'italian'
- Without the hint
-
SELECT surname FROM drivers WHERE nationality = 'Italian'; - Result
- 97 rows. Correct.
- With the hint
-
SELECT surname FROM drivers WHERE nationality = 'italian'; - Result
- No rows. SQLite compares text exactly, capital letters included, so nothing matches.
677 not SQLite The hint uses YEAR(), which SQLite does not have.
- Question
- How many posts did Jay Stevens have in 2010?
- Hint
- DisplayName = 'Jay Stevens'; in 2010 refers to YEAR(CreationDate) = 2010;
- Without the hint
-
SELECT COUNT(*) FROM posts JOIN users ON posts.OwnerUserId = users.Id WHERE users.DisplayName = 'Jay Stevens' AND STRFTIME('%Y', posts.CreaionDate) = '2010'; - Result
- 3. Correct.
- With the hint
-
SELECT COUNT(*) FROM posts JOIN users ON posts.OwnerUserId = users.Id WHERE users.DisplayName = 'Jay Stevens' AND YEAR(posts.CreaionDate) = 2010; - Result
- Error: no such function.
1490 integer division Following the hint's formula in order rounds the answer down to 0.
- Question
- How many percent of LAM customer consumed more than 46.73?
- Hint
- Percentage of LAM customer consumed more than 46.73 = (Total no. of LAM customers who consumed more than 46.73 / Total no. of LAM customers) * 100.
- Without the hint
-
SELECT CAST(SUM(CASE WHEN T2.Consumption > 46.73 THEN 1 ELSE 0 END) AS REAL) * 100 / COUNT(T1.CustomerID) AS Percentage FROM customers AS T1 JOIN yearmonth AS T2 ON T1.CustomerID = T2.CustomerID WHERE T1.Segment = 'LAM'; - Result
- 98.53. Correct.
- With the hint
-
SELECT (SUM(CASE WHEN T2.Consumption > 46.73 THEN 1 ELSE 0 END) / COUNT(T1.CustomerID)) * 100 AS Percentage FROM customers AS T1 JOIN yearmonth AS T2 ON T1.CustomerID = T2.CustomerID WHERE T1.Segment = 'LAM'; - Result
- 0. Dividing one whole number by another, SQLite drops the fraction before the * 100.
788 contradicts the key The hint's formula and the answer key compute different percentages.
- Question
- How many percent of female heroes were published by Marvel Comics?
- Hint
- percent = MULTIPLY(DIVIDE(SUM(gender = 'Female' WHERE publisher_name = 'Marvel Comics'), COUNT(publisher_name = 'Marvel Comics')), 100); ...
- Without the hint
-
SELECT CAST(SUM(CASE WHEN T2.publisher_name = 'Marvel Comics' THEN 1 ELSE 0 END) AS REAL) * 100 / COUNT(*) AS percentage FROM superhero AS T1 JOIN publisher AS T2 ON T1.publisher_id = T2.id JOIN gender AS T3 ON T1.gender_id = T3.id WHERE T3.gender = 'Female'; - Result
- 54.95: the share of female heroes who are Marvel. Correct.
- With the hint
-
SELECT CAST(SUM(CASE WHEN T1.gender = 'Female' THEN 1 ELSE 0 END) AS REAL) * 100 / COUNT(*) AS percent FROM gender AS T1 JOIN superhero AS T2 ON T1.id = T2.gender_id JOIN publisher AS T3 ON T2.publisher_id = T3.id WHERE T3.publisher_name = 'Marvel Comics'; - Result
- 28.68: the share of Marvel heroes who are female, exactly what the hint asks for. Wrong by the answer key.
What comes next
Part 3 keeps the model fixed and changes the prompt instead: column descriptions, example values, values from the question found in the database, and asking the model to reason before it answers. One at a time, to see what each is worth.
BIRD is by Jinyang Li and colleagues (arXiv 2305.03111) and is shared under CC BY-SA 4.0. The questions, SQL and rows quoted here are from it.🥹 If this helped, a like means a lot.