On this page

Lexicons, motifs and joint margins

Three declarations added in 0.9.6.46 through 0.9.6.50. Each one closes a gap where the engine used to produce something plausible instead of something stated.

semantic: vocabulary that keeps growing#

A text column used to draw from a fixed pool. Over thirty thousand rows that pool runs out and the column starts repeating, which is the fastest way for synthetic data to give itself away.

semantic names what a column means, and the engine resolves it to a generative lexicon: a small head of real, high-frequency values drawn with Zipfian weight over rank, plus composition over morpheme slots for everything past the head.

Python
schema = {
    "port_calls": {
        "__rows__": 30_000,
        "id": {"type": "integer", "primary_key": True},
        "vessel": {"type": "text", "semantic": "vessel_name"},
        "master": {"type": "text", "semantic": "person_name"},
    },
}

Built-in lexicons: person_name, company_name, vessel_name, medical_procedure. Inspect one with:

Python
from misata.lexicon import get_spec
spec = get_spec("vessel_name")
print(spec.description, spec.effective_capacity())

Effective capacity, not raw capacity#

effective_capacity() is the number that matters. A pattern drawn 4% of the time contributes 4% of the draws however many values it could form, so a column duplicates at the rate of the pattern that saturates first. Raw capacity overstates by up to 300x.

Feasibility refuses a column whose row count exceeds what its lexicon can carry, before generating anything, using a per-type rows_per_distinct. Repetition is a property of the type, not a defect: clinical coding genuinely concentrates on a handful of procedures, so thirty thousand rows over a few thousand distinct values is what real data looks like, while thirty thousand customers sharing two thousand names is not.

Locale still wins#

Person names are region-specific and the locale pack gets them right. Lexicons marked locale_sensitive step aside for any locale other than en_US, so locale="ja_JP" still returns 鈴木 くみ子.

graph_motifs: the patterns worth detecting exist on purpose#

dag_edges guarantees a graph with no cycles. That is the right default and the wrong dataset for anyone building a detector, because the shapes worth finding are exactly the ones a DAG forbids.

graph_motifs rewrites a declared fraction of an edge table into rings, fan-in, fan-out, scatter-gather and chains at an exact mix, each labelled with a case id, leaving every other edge as the DAG put it.

Python
schema = {
    "accounts": {"__rows__": 5_000, "id": {"type": "integer", "primary_key": True}},
    "transfers": {
        "__rows__": 80_000,
        "src": {"type": "integer"}, "dst": {"type": "integer"},
        "amount": {"type": "float", "min": 10, "max": 90_000},
    },
    "__graph_motifs__": [{
        "name": "laundering", "table": "transfers",
        "from_column": "src", "to_column": "dst",
        "node_table": "accounts", "node_key": "id",
        "rate": 0.02,
        "shares": {"cycle": 0.4, "fan_in": 0.25, "fan_out": 0.2, "scatter_gather": 0.15},
        "benign_rate": 0.02,
        "benign_shares": {"cycle": 0.5, "fan_in": 0.5},
        "flag_column": "is_suspicious",
    }],
}

Three columns come back on the edge table. motif names the shape (cycle, fan_in, fan_out, scatter_gather, chain, or "" for a background edge), motif_case groups the edges belonging to one instance, and flag_column, if you name one, is the label: True for the flagged motifs and False for everything else, including the benign ones. Without it a benign ring and a flagged ring are indistinguishable, which defeats the point of declaring them.

The property that follows is exact rather than statistical:

the subgraph of edges carrying no case id is acyclic

So every cycle in the output belongs to a case somebody declared, and an accidental pattern cannot exist rather than merely being unlikely. A detector run against it cannot produce an unexplained hit.

benign_shares declares hard negatives: real motifs of the same shapes, labelled legitimate. A detector is then measured on telling a ring from an innocent loop rather than on finding loops.

Feasibility refuses a node pool too small to lay out the widest motif without repeating a node inside one case.

joint_distributions: several margins, all at once, exactly#

Two declared margins used to be satisfiable one at a time and silently inconsistent together. This solves for the unique maximum-entropy table consistent with every declared margin, by iterative proportional fitting.

Python
schema = {
    "accounts": {
        "__rows__": 50_000,
        "region": {"type": "string", "enum": ["emea", "apac", "amer"]},
        "tier": {"type": "string", "enum": ["free", "pro", "enterprise"]},
    },
    "__joint_distributions__": [{
        "name": "region_by_tier", "table": "accounts",
        "margins": {
            "region": {"emea": 0.42, "apac": 0.31, "amer": 0.27},
            "tier": {"free": 0.70, "pro": 0.22, "enterprise": 0.08},
        },
    }],
}

Margins that cannot all hold are refused up front, with both declarations and the arithmetic named, rather than one being quietly dropped.

Two-way tables integerise with both margins exact, which is always solvable. Three-way and above preserve the grand total exactly and say so rather than implying more, because integer margin preservation above two dimensions has no such guarantee.

Pass emphasis to bias the interior of the table towards a known association, or forbidden to zero out combinations that cannot occur.

Last updated Edit this page.md