Misata: Outcome-Conformant Synthetic Data Generation

Muhammed Rasin

For data-agent evaluation and RL environments

Benchmark databases generated from the answer key

A VLDB 2026 audit found 52.8 percent of BIRD Mini-Dev answer keys are wrong, because benchmarks annotate answers onto databases after the fact. An evalpack inverts the order: the answers come first, the database is generated to satisfy them, and every shipped question is verified by independent execution. Re-verify any pack yourself in about thirty seconds.

pip install duckdb

python pack/verify.py

# 35/35 verified exactly

Step 1

Declare the ground truth

Aggregate curves, rates, and foreign-key constraints. The answer key is a specification, not an annotation someone might get wrong.

Step 2

Generate the database to match

A deterministic solver emits multi-table data that satisfies the declaration exactly. No LLM in the data path, fully seeded.

Step 3

Verify with an independent engine

Every question's gold SQL runs against the shipped CSV files with DuckDB, which shares no code with the generator. Anything inexact is dropped and logged, never shipped.

Step 4

Rotate whenever you like

Spec plus seed fully determine the pack. A new seed replaces 99.2 percent of the rows and none of the answer key, so agents cannot memorize the environment.

See one, or build your own

The public demonstration pack (73,629 orders, 35 verified questions, certificate and manifest included) lives on GitHub with all the experiment harnesses from the paper. Building your own takes one command from any schema:

pip install "misata[evalpack]"

misata evalpack --config your_schema.yaml -o your_pack

Demonstration pack on GitHub Read the write-up

Request a custom pack

I build verified packs to spec: your domain, your tables, your declared outcomes, at whatever scale the evaluation needs, usually within 48 hours. CRM pipelines whose books reconcile, marketplaces with declared GMV and refund rates, ledgers where every journal entry balances to the cent. Every pack ships with its certificate, manifest, and standalone verifier. The engine is MIT; the service is how teams without time get exactly the environment they need.

Tell me the domain, the tables, the outcomes to declare, the scale, and the deadline:

Request via GitHub issue Or email directly

Method paper: arXiv:2606.08736 and the answer-key-first construction paper (in review). Every claim on this page is machine-checkable from the artifacts; where the approach fails is documented honestly in the limitations page.