Realism benchmark

Misata's default mode never sees real data, so "realistic" cannot be shown the usual way: fit a model to a dataset, then score it against that dataset. This benchmark asks the question a user actually faces. Knowing only the shape of a business (its tables, its column names and types, roughly how many rows), how close does each tool get to what real data of that kind looks like?

Two public datasets, each split in half:

• Olist: real orders from a Brazilian marketplace (2016–2018), 30,000 orders sampled, with customers, products and payments.
• NYC taxis: the seaborn sample of March 2019 trips, 6,389 rides.

Contestants:

Everything is scored against the held-out test half, on scale-free metrics so a blind generator is not punished for not knowing the currency:

Results#

Misata 0.9.7, seed 7. Reproduce with python -m benchmarks.realism_bench --cache .bench_cache.

Detection AUC over five seeds#

Seeds 7, 11, 23, 42 and 99: mean, with the lowest and highest run. The other metrics move by 0.01 or less between seeds; every one is in benchmarks/results/summary.json.

• Olist
• NYC taxis

Show the numbers

Generator	Olist	NYC taxis
real_train (noise floor)	0.5	0.503
misata_mimic (fitted)	0.537	0.543
misata_schema (blind)	0.629	0.75
sdv_copula (fitted)	0.711	0.675
misata_story (blind)	0.73	0.801
faker_script (blind)	0.783	0.788

Source: benchmarks/results/summary.json, realism.datasets.*.detect_auc.mean

Olist (30,000 orders)#

NYC taxis (6,389 trips)#

What it says#

On e-commerce, blind Misata beats the script and approaches a model fitted to the data. With table and column names and the domain, misata_schema is harder to tell from real orders than SDV fitted on 15,000 of them (detection AUC 0.63 against 0.71, over five seeds), gets the shape of order amounts closer than SDV (0.05 vs 0.07), and matches real product popularity almost exactly (ΔGini 0.03), which the uniform script misses by five times as much. One qualification: the e-commerce amount prior was set from Olist's public data when Misata's priors were built, so the amount row is not a blind result. Hours, weekdays and fan-out are.

On taxis, the hour rhythm is now right; fares are not. Declaring the domain as transport gives the night-heavy demand curve ride data has (hour TVD 0.08, better than SDV fitted on the data at 0.19), which puts blind Misata ahead of the script (AUC 0.75 vs 0.79) but behind SDV fitted on the real trips (0.68). Fares are still drawn from a generic money shape that is wider than taxi totals (amount KS 0.25, worse than the script's 0.22). That is the remaining miss here, and it is left in rather than fixed with a taxi-specific fare prior fitted to the same public data.

Customer fan-out is better, not solved. In this Olist sample nearly every buyer buys once (30,000 orders from 29,651 customers). Declaring the domain as marketplace makes person-like parents (customers, buyers, guests) mildly weighted while products stay concentrated, which cut the gap from 0.39 to 0.26. Declaring min_children: 1 on the relationship, which says what this sample says (every customer in it has ordered), closes it: ΔGini 0.000. That declaration did not work before 0.9.7 for child tables larger than one 10,000-row batch; it now counts coverage across batches.

mimic is the strongest fitted generator here. It beats SDV on every metric except weekday profile and category balance on taxis, and is close to indistinguishable on both datasets (AUC 0.54).

The tells check calibrates on real data. Both real datasets score 1.0 on realism_report.

Changes between runs, and why#

The first published run (0.9.6.60 defaults) found five problems, and the fixes are in 0.9.7. To keep the benchmark honest about what it tests:

• Bugs, fixed generally: mimic dropped the time of day; story *_at columns were dates; the fan-out tell flagged real data; product names were not reproducible across processes. None of these fixes look at the benchmark data.
• Domain knowledge added after the benchmark exposed its absence: a night-heavy hour curve for transport/taxi/mobility/nightlife domains, and mild person fan-out for marketplace/travel/realestate. These are general facts about those businesses, not parameters fitted to these samples, but they were added because these samples showed they were missing. Treat the taxi hour row and the Olist customer row as validation of a fix, not as a blind first measurement.
• The Olist blind schema now declares marketplace (it declared ecommerce before), because that is what Olist is. With the same declaration as before, every number but customer fan-out is unchanged.

A correction to the detection metric#

The first 0.9.7 runs reported higher detection AUCs (Olist schema 0.74, taxi schema 0.91, taxi mimic 0.66). Running five seeds showed why: the real train half scored as high as 0.89 against the real test half on some seeds, which is impossible for a fair test. Amounts are discrete (prices repeat), and dividing each sample by its own median put the two real halves on slightly different grids that the classifier learned to tell apart. The feature is now rounded to one decimal of log-ratio, real-vs-real scores 0.50 on every seed, and every contestant's AUC above is from the corrected metric. The other metrics did not change.

Caveats#

• Two datasets, both transactional. This is evidence about orders and trips, not about every domain.
• The detection classifier sees three scale-free features. A classifier with more features (text, cross-table joins) would separate every contestant more easily.
• Blind contestants get the row counts and, for the schema variant, the category labels. They never see shares, ranges or any row.
• Five seeds. A seed changes which 30,000 Olist orders are sampled, the train/test split and every generator's seed; it cannot add datasets.

Related#

• Realism report: the tells check behind the tells score, for any generator
• Validity benchmark: keys and CHECK rules rather than shapes
• Mimic: the fitted contestant, which is not anonymous
• Limitations: the taxi fare miss and other known gaps