On this page
Benchmarks
Three benchmarks, three different questions. Each number on this page comes from a file in the repository and has a command that reproduces it.
| Benchmark | Question | Reproduce |
|---|---|---|
| Validity | Does the data obey its own schema's keys and CHECK rules? | python -m benchmarks.validity_bench |
| Realism | How close does it get to real data of the same kind, without seeing any? | python -m benchmarks.realism_bench --seed N |
| The Gauntlet | Does it survive a reviewer running arbitrary joins and group-bys? | python -m benchmarks.gauntlet |
The validity and realism numbers are collected in
benchmarks/results/summary.json.
Validity: zero violations, by construction#
Five SQL schemas, each generated from its DDL with nothing else configured, and checked by a pandas checker that shares no code with Misata.
- Misata from DDL
- Faker script
- SDV HMA
Show the numbers
| Schema | Misata from DDL | Faker script | SDV HMA |
|---|---|---|---|
| saas_billing | 0 | 36,150 | could not run |
| org_chart | 0 | 15,014 | 4,807 |
| marketplace_m2m | 0 | 42,387 | 5,557 |
| hotel_bookings | 0 | 50,416 | 3,055 |
| retail_chain_6_levels | 0 | 24,025 | could not run |
Source: benchmarks/results/summary.json, validity.totals; Misata 0.9.7, seed 7
SDV's open-source multi-table model refuses schemas with more than five tables or foreign keys more than two levels deep, so it could not run two of the five. Full results and method.
Realism: better than a script, behind a fitted model on taxis#
A classifier tries to tell generated rows from held-out real rows; 0.5 means it cannot. Blind contestants never see a real row.
- Olist
- NYC taxis
Show the numbers
| Generator | Olist | NYC taxis |
|---|---|---|
| misata_mimic (fitted) | 0.537 | 0.543 |
| misata_schema (blind) | 0.629 | 0.75 |
| sdv_copula (fitted) | 0.711 | 0.675 |
| faker_script (blind) | 0.783 | 0.788 |
Source: benchmarks/results/summary.json, realism.datasets.*.detect_auc.mean
On Olist orders, blind Misata is harder to detect than SDV fitted on the
real orders. On taxi trips it beats the Faker script but not SDV fitted on
the trips, because blind fares are drawn from a generic money shape.
mimic scores best and is fitted to the real rows, so it is not anonymous.
Full results, every metric and the caveats.
The Gauntlet: 136 SQL assertions#
A 13-table relational schema and 136 independent SQL assertions, each a query returning the number of violating rows, run by DuckDB against the generated tables. The assertions cover keys, value domains, temporal order, status gates, parent totals that equal their child facts (including across several hops), denormalised copies, geography, arithmetic, degenerate distributions, lifecycles, missingness and late arrivals. Aggregate checks are exact to the cent, in SQL.
Show the numbers
| Generator | Assertions held |
|---|---|
| Misata | 136 |
| Faker script | 78 |
Source: CHANGELOG.md, 0.9.8, "Declared values hold"
The Faker script is a careful one: foreign keys sampled from parent ids,
real formats, line totals computed as quantity times price. What it does not
do is build a constraint solver by hand, so aggregates, status gates,
temporal order and geography consistency fail. Reproduce with
python -m benchmarks.gauntlet and
python -m benchmarks.gauntlet_compare --tool faker. The harness can also
score SDV (--tool sdv), but no SDV score is published because it has not
been run.
Reproducibility#
python -m benchmarks.golden regenerates a set of story, dict and DDL
schemas and compares their misata.fingerprint hashes; CI requires the same
hashes on Linux, macOS and Windows.
Related#
- Limitations: what none of these benchmarks covers
- Research benchmarks: the offline harness against a hand-written Faker baseline
- Faker vs SDV vs Misata: choosing between them
Prefer generating data visually?
Misata Studio lets you turn natural language prompts into relational schemas with exact outcome invariants, live preview tables, and one-click database seeding without writing code.