Anyone can generate 11 tables. Can the data survive 126 SQL assertions?
Every synthetic data tool produces something that looks right in a preview. The interesting question is what happens when a reviewer runs arbitrary JOINs and GROUP BYs against it. So we built an 11-table e-commerce schema with an M:N junction and a diamond dependency, wrote 126 assertions about what must be true of it, and had DuckDB check them. DuckDB had no part in generating the data, so the verifier and the generator share no code.
The two tools pass exactly the same 72 assertions. Misata passes 54 more, and there is no assertion Faker passes that Misata fails. Every one of those 54 differences sits in one of 8 categories: temporal causality, status implications, reconciliation, geographic consistency, lifecycle, missingness, late arrival, grid and duplicates. In other words, Faker is not bad at generating values. It is excellent at values. It simply has no concept of a fact that spans two rows.
| Category | Faker | Misata |
|---|---|---|
Structural Primary keys unique and not null, every foreign key resolving to a parent that exists. | 32/32 | 32/32 |
Domain Value ranges, formats, and enum membership. Is a price positive, is a state a real state, is a zip five digits. | 20/20 | 20/20 |
Temporal causality No child event predates the parent that caused it. An order before its customer signed up cannot happen, and a line item cannot contain a product invented after the order. | 1/10 | 10/10 |
Status implications A status gates its dependent columns. An active subscription has no cancellation date; a cancelled order has no payments. | 0/10 | 10/10 |
Reconciliation Parent aggregates equal what child rows actually sum to, including through two joins, and child money never exceeds the parent's. | 0/8 | 8/8 |
Diamond dependency A denormalized copy agrees with its source. The price on a line item is the price of the product it points at. | 2/2 | 2/2 |
Geographic consistency One state per city, one city per zip, and the same city agreeing across tables. | 0/5 | 5/5 |
Derived arithmetic Computed columns satisfy the formula they were declared with. | 2/2 | 2/2 |
Distribution sanity The data is not degenerate: spread in the values, a heavy tail in the child counts, some parents with no children at all. | 10/10 | 10/10 |
Lifecycle A row's state implies a legal, ordered history: every timestamp on its path present and in order, every one off its path null. | 3/11 | 11/11 |
Missingness Values go missing for a declared reason rather than at a flat rate, which is the pattern real data has and MCAR does not. | 0/4 | 4/4 |
Late arrival Some events land in a later partition than they happened, at the declared rate and within the declared bound. Every incremental model assumes this never happens. | 0/5 | 5/5 |
Grid and duplicates Timestamps sit on the clock a human would use, and an exact number of rows are copies of another row, so dedupe logic has something to find. | 2/7 | 7/7 |
| Total | 72/126 | 126/126 |
Where the 54 differences actually are
These categories are the ones that need a generator to reason about more than one row at a time. Violating-row counts are from the real run, not illustrations.
Temporal causality Misata 10/10, Faker 1/10
- orders never precede customer signupMisata: pass · Faker: 806 violating rows
- subscriptions never precede customer signupMisata: pass · Faker: 248 violating rows
- support tickets never precede customer signupMisata: pass · Faker: 254 violating rows
- payments never precede their orderMisata: pass · Faker: 1,409 violating rows
- shipments never precede their orderMisata: pass · Faker: 925 violating rows
- returns never precede their orderMisata: pass · Faker: 143 violating rows
- order_items never reference a product created after the orderMisata: pass · Faker: 641 violating rows
- tickets resolved after they were createdMisata: pass · Faker: 244 violating rows
- subscriptions cancelled after they startedMisata: pass · Faker: 82 violating rows
- no order predates the shop's first customerMisata: pass · Faker: pass
Status implications Misata 10/10, Faker 0/10
- cancelled subscriptions have cancelled_atMisata: pass · Faker: 96 violating rows
- active subscriptions have no cancelled_atMisata: pass · Faker: 125 violating rows
- past_due subscriptions have no cancelled_atMisata: pass · Faker: 15 violating rows
- resolved tickets have resolved_atMisata: pass · Faker: 148 violating rows
- closed tickets have resolved_atMisata: pass · Faker: 60 violating rows
- open tickets have no resolved_atMisata: pass · Faker: 84 violating rows
- pending tickets have no resolved_atMisata: pass · Faker: 65 violating rows
- shipments only for shipped/completed/returned ordersMisata: pass · Faker: 237 violating rows
- returns only for returned/return_pending ordersMisata: pass · Faker: 269 violating rows
- cancelled orders have no paymentsMisata: pass · Faker: 128 violating rows
Reconciliation Misata 8/8, Faker 0/8
- orders.total_amount = sum(order_items.line_total), to the centMisata: pass · Faker: 2,500 violating rows
- customers.order_count = count(orders), exactlyMisata: pass · Faker: 451 violating rows
- customers.lifetime_value = sum(payments via orders), to the centMisata: pass · Faker: 500 violating rows
- grand total: sum(orders.total_amount) = sum(order_items.line_total)Misata: pass · Faker: 1 violating rows
- grand total: sum(customers.lifetime_value) = sum(payments.amount)Misata: pass · Faker: 1 violating rows
- grand total: sum(customers.order_count) = count(orders)Misata: pass · Faker: 1 violating rows
- refund never exceeds the order totalMisata: pass · Faker: 52 violating rows
- payments per order never exceed the order totalMisata: pass · Faker: 841 violating rows
Geographic consistency Misata 5/5, Faker 0/5
- customers: one state per cityMisata: pass · Faker: 5 violating rows
- customers: one city per zipMisata: pass · Faker: 1 violating rows
- addresses: one state per cityMisata: pass · Faker: 23 violating rows
- addresses: one city per zipMisata: pass · Faker: 3 violating rows
- city/state pairs agree across customers and addressesMisata: pass · Faker: 29 violating rows
Lifecycle Misata 11/11, Faker 3/11
- orders.status is a declared lifecycle stateMisata: pass · Faker: pass
- every order has placed_at (the initial state)Misata: pass · Faker: pass
- shipped_at present exactly when shipped is on the pathMisata: pass · Faker: 865 violating rows
- completed_at present exactly when completed is on the pathMisata: pass · Faker: 1,152 violating rows
- cancelled_at present exactly when cancelledMisata: pass · Faker: 340 violating rows
- cancelled orders never carry a shipment timestampMisata: pass · Faker: 110 violating rows
- path order holds: placed <= shipped <= completedMisata: pass · Faker: 1,230 violating rows
- cancellation never precedes placementMisata: pass · Faker: 117 violating rows
- the whole chain postdates order_dateMisata: pass · Faker: 1,289 violating rows
- returned orders carry the full upstream chainMisata: pass · Faker: 100 violating rows
- every declared state occurs (the machine is exercised)Misata: pass · Faker: pass
Missingness Misata 4/4, Faker 0/4
- income is missing for ~40% of the 18-24 band, as declaredMisata: pass · Faker: -1 violating rows
- income is missing for ~5% of everyone else, as declaredMisata: pass · Faker: -1 violating rows
- missingness is conditional, not uniform (MNAR not MCAR)Misata: pass · Faker: -1 violating rows
- the column is not entirely null and not entirely presentMisata: pass · Faker: -1 violating rows
Late arrival Misata 5/5, Faker 0/5
- nothing is ingested before it happenedMisata: pass · Faker: -1 violating rows
- every order has an ingest timestampMisata: pass · Faker: -1 violating rows
- ~4% of orders arrive a day or more late, as declaredMisata: pass · Faker: -1 violating rows
- no delay exceeds the declared 3-day boundMisata: pass · Faker: -1 violating rows
- late arrivals actually exist (the path is exercised)Misata: pass · Faker: -1 violating rows
Grid and duplicates Misata 7/7, Faker 2/7
- every ticket opens on the declared 15-minute gridMisata: pass · Faker: 741 violating rows
- no ticket carries seconds or sub-secondsMisata: pass · Faker: 800 violating rows
- every ticket opens inside declared business hoursMisata: pass · Faker: 541 violating rows
- the grid is exercised (more than one distinct slot in use)Misata: pass · Faker: 1 violating rows
- exactly 24 duplicate tickets, as declared (3% of 800)Misata: pass · Faker: 1 violating rows
- duplicated tickets still carry distinct idsMisata: pass · Faker: pass
- duplication did not change the declared row countMisata: pass · Faker: pass
Where both tools pass, and why that matters
structural, domain, diamond dependency, derived arithmetic, distribution sanity: both score full marks on all 66 assertions in these categories. This is the part of the benchmark that keeps it honest. A benchmark where the author wins every category is usually a benchmark designed to be won. Foreign key integrity, value formats, real state codes, derived formulas, and distribution shape are all solved problems if you write the script carefully, and Faker solves them.
What happens when there is nothing left to fail
Nothing is red today. The last one was foreign key sampling without temporal eligibility, which let an order line reference a product created after the order was placed. It sat here in the open for weeks, failing visibly in every run, and it was fixed in 0.9.1.
The mechanism it used has not been removed, because it is the part that keeps this page honest. An assertion written before the capability exists goes on a known-red list with a roadmap note. Continuous integration treats it as an expected failure, fails the build on any other regression, and also fails the build if a known-red starts passing without being promoted out. A full score is a fact about today, not a claim about the suite.
An acceptance test that quietly shrinks to fit the product is not an acceptance test.
Method, stated plainly
The schema. 11 tables and 16,128 rows: customers, addresses, categories, products, orders, order_items as a true M:N junction pointing at both orders and products, payments, shipments, returns, subscriptions, support_tickets.
The verifier. Every assertion is a SQL query returning a count of violating rows, run by DuckDB against the emitted frames. Zero is a pass. Equality checks on money carry a one cent tolerance in the SQL itself, so exact means exact in SQL, not in the generator's own bookkeeping.
The Faker script. A careful but ordinary script. Foreign keys are sampled from real parent ids, formats come from Faker's own providers, state codes use every fairness flag Faker offers, prices follow the same declared formula, and line totals are computed rather than drawn. What it does not do is hand-build a constraint solver, because at that point you would not be benchmarking Faker, you would be writing the correctness layer yourself. That is precisely the work being measured.
Tools not scored. SDV has a ready code path in the harness but was not run here, so it gets no number rather than an estimated one. Seedfast cannot be scored offline at all: it is a closed CLI requiring an account and a live Postgres database. Both are runnable by anyone who has them, against the same 126 assertions.
Run it yourself
The harness is in the repository, MIT licensed. Nothing here needs to be taken on trust.
git clone https://github.com/rasinmuhammed/misata.git cd misata pip install -e ".[dev]" python -m benchmarks.gauntlet # 126/126 python -m benchmarks.gauntlet_compare --tool faker # 72/126
Measured 2026-07-27 on misata 0.9.1, Faker 40.11.1. Every number on this page came from those two commands. When the engine changes, the page is regenerated from a fresh run rather than edited.