How to test a data pipeline before you have data

Most data pipeline tests check that the code runs. Very few check that the numbers are right, because to check the numbers you need data whose correct answer you already know. Production data rarely comes with a known answer, and staging is usually empty on day one. So the transform ships, and the first time anyone verifies the output is in a dashboard three weeks later.

There is a better way, and it flips the usual order. Instead of generating data and hoping the pipeline handles it, you declare the answer first, generate data engineered to produce that answer, and then assert your pipeline reproduces it. This is known-answer testing, and it gives you a test that can actually fail for the right reason.

The idea in one example

Say your pipeline aggregates orders into monthly revenue, and you want to prove it rolls up correctly. Declare the truth up front:

import misata

schema = misata.from_dict_schema({
    "orders": {
        "id":       {"type": "integer", "primary_key": True},
        "amount":   {"type": "float", "min": 5, "max": 500},
        "placed_at": {"type": "date", "min": "2025-01-01", "max": "2025-12-31"},
    },
    # Declare the monthly revenue curve the data must sum to, exactly.
    "__outcome_curves__": [{
        "table": "orders", "column": "amount", "time_column": "placed_at",
        "time_unit": "month", "value_mode": "absolute",
        "curve_points": [
            {"month": 1, "target_value": 50000},
            {"month": 12, "target_value": 200000},
        ],
    }],
}, seed=42)

tables = misata.generate_from_schema(schema)

Now January's orders sum to exactly 50000 and December's to exactly 200000, with a smooth interpolation between. Run your pipeline over this data and assert the output matches. If your GROUP BY is wrong, if a join drops rows, if a timezone bug shifts a month boundary, the assertion fails. That is a test with teeth.

Why synthetic data is the right fit here

The reason this works is that Misata generates from a specification, not from a sample. You are not imitating a real dataset; you are constructing one whose aggregates are known by construction. That has three properties known-answer testing needs:

• Exactness. The monthly totals are not approximately right, they are right to the cent. The mechanism is closed-form conditional sampling, formalized in an arXiv paper. Imitation synthesizers trained on real data miss declared monthly aggregates badly; see the benchmark for the numbers.
• Referential integrity. If your pipeline joins orders to customers, every order points at a real customer and the join never silently drops rows. Misata reports orphan counts per relationship so you can assert on integrity too.
• Reproducibility. The same seed produces identical bytes on any machine, so the test is deterministic in CI. No flaky fixtures.

Beyond revenue: rates, and the fraud-model trick

The same pattern works for rates. If you are testing a fraud-detection pipeline, declare the fraud rate and Misata produces a labeled dataset that hits it exactly. You made up the answer, so you can prove the detector finds it. That is the difference between a test that checks plumbing and a test that checks correctness. See synthetic data for fintech for the full setup.

Where this fits in your test suite

Known-answer tests are not a replacement for testing against real data once you have it. They are the layer that exists before real data does, and the layer that stays fast and deterministic forever after. A good split:

• Unit tests for transform logic on tiny hand-built inputs.
• Known-answer tests for aggregate correctness on generated data with a declared truth.
• Contract tests against real data samples once production exists.

The first two need no real data and run in milliseconds. Start there.

Try it

pip install misata

Or design the schema on a canvas and generate in your browser with Misata Studio. If you are choosing between tools, the comparison pages lay out how this differs from Faker, SDV, and Gretel.