Verified eval databases
Text-to-SQL benchmarks are wrong about half the time. Declare the answer first and the answer key cannot be wrong by construction.
Anyone training or evaluating a data agent, and anyone who needs a reward signal they can trust
The problem
Reward signal for SQL reasoning is measurably corrupted, and the corruption is documented rather than suspected. A CIDR 2026 audit from UIUC found 52.8% of BIRD Mini-Dev annotations wrong, and 66.1% of Spider 2.0-Snow. Correcting them moved leaderboard positions by up to nine places.
The largest synthetic corpus, SynSQL-2.5M, is 86% fully correct by its own human evaluation, which leaves roughly 356,000 wrong examples shipped as ground truth.
The same group separately showed that noisy data is destructive to reinforcement learning with verifiable rewards. To get clean data for that paper they hand-corrected 600 BIRD instances, one at a time.
Why the usual answer falls short
The most common error class in that audit, at 57.8%, is the annotator misunderstanding the schema or the data they were writing a query against. That is not carelessness. It is what happens when a human reads an unfamiliar database and guesses what a query should return.
Every existing fix attacks the problem from the same side: better review, more annotators, an agent that hunts for errors after the fact. Nobody changes the direction the data was built in.
So the errors keep coming from the same place, because the answer is always derived from a database somebody had to interpret.
Head to head
Someone reads an unfamiliar database and writes the query they think answers the question. Measured error: 52.8% on BIRD Mini-Dev, 66.1% on Spider 2.0-Snow.
Nobody reads the data. The answer is stated first and the database is solved backward to satisfy it, so misreading a schema is not a risk you mitigate, it is a state that cannot occur.
A model writes the question, the query and the data together. SynSQL-2.5M reports 86% fully correct, so roughly one in seven rewards is wrong.
A deterministic solver, no model in the data path, and every shipped question re-executed against the written files by an engine that shares no code with the generator.
Only what passed is published. The rejection set, if one existed, stays private.
All 12,150 refused candidates ship with the corpus, each with its declared answer, its observed answer and the reason. A corpus claiming verified answers should show what its verifier threw out.
What was declared
Seven financially coherent schema families, each with exact monthly totals, exact rates, exact category splits and, for the ledger family, double-entry books that balance to the cent. Questions are derived from those declarations, never from the generated rows.
Primitives used: outcome_curves, rate_curves, group_shares, balanced_ledger. Seed 20260904, engine 0.9.6.45.
What came out, and how it was checked
| Questions shipped, every one independently verified | 109,306 |
| Databases, tables, and data rows | 1,400 / 4,400 / 23,738,983 |
| Gold SQL re-executed from the published files | 109,306 / 109,306 scored 1.0 |
| Cross-checked in pandas, a different engine with no SQL | 1,450 / 1,450 agreed |
| Candidates refused, logged, and published anyway | 12,150 |
| Score from always answering the most common value | 1.83% |
The last row is the one that matters. A strongly connected component of one node means no cycle exists anywhere outside a declared case, so an accidental pattern is not unlikely, it is impossible.
Running a detector against it
How it compares to what the field currently trains and evaluates on:
| BIRD Mini-Dev, human annotated | 52.8% of annotations wrong |
| Spider 2.0-Snow, human annotated | 66.1% wrong |
| SynSQL-2.5M, LLM generated | ~14% wrong, by their own eval |
| This corpus, declared then verified | 0 shipped without an exact match |
The comparison is narrower than it looks and the dataset card says so. These questions are generated, so they probe a narrower band of reasoning than human-authored ones. The contribution is a clean reward signal, not harder questions, and it is not a BIRD replacement.
What this does not claim
- Not a BIRD or Spider replacement. Generated questions are structurally simpler than human-authored ones. The contribution is the reward being right, not the question being hard.
- Question phrasing repeats. A paraphrase bank gives 1,055 distinct phrasings, up from 188, but across 109,306 questions a given phrasing still recurs about a hundred times. Train on it for SQL construction, not for breadth of language understanding.
- Ambiguity is mitigated, not eliminated. Templated phrasing removes most wording ambiguity. Wording is still wording.
- Free text is a structural placeholder and is not realistic. Nothing here tests language realism and it should not be used to.
Questions
- What does "the answer key cannot be wrong" actually mean?
- The expected answer exists before the database does. The generator solves for rows that satisfy it, then DuckDB executes the gold SQL against the written files and the question only ships if the result matches exactly. An answer key derived from data somebody interpreted can be wrong; one the data was built to satisfy cannot be, for the classes of error that come from misreading a schema.
- Why publish the questions you rejected?
- Because a corpus that claims verified answers should show what its verifier threw out. Almost all 12,150 are rate anchors, where a declared rate like 2% is not exactly reachable over an integer number of rows. None of the widely used corpora report a rejection set at all.
- Is it contaminated?
- It cannot be memorised in advance, because every instance is generated from a seed and a fresh, statistically comparable instance can be produced at any time by changing it. Fixed benchmark files decay as they leak into training sets. This one does not have to.
- How do I plug it into a training loop?
- One function. The corpus ships a reference grader whose only dependency is duckdb: reward(question, predicted_sql) returns 1.0 or 0.0, exact match against a declared answer. It also emits prompt-ready CREATE TABLE text for the database behind each question.
- Where do I get it?
- It is published on Hugging Face at rasinmuhammed/verified-sql-rewards, under Apache 2.0, with the dataset card stating the limits up front rather than in a footnote.
Build this yourself
Everything above is a schema and a seed. Install the engine, declare what has to be true, and check the result with whatever you trust.