Research · phase P0 Thesis not validated. This page reports a measurement, not a product.

Sanctions watchlists are written in Latin script.
The trade corridor under scrutiny is written in Cyrillic.

Between those two facts sits a matching problem that has no canonical answer: one Cyrillic name has many legitimate Latin spellings, and no standard governs which one a record used. Tamga is an open benchmark that measures how badly commodity matchers actually handle it — built to be capable of killing its own premise.

Щербаковone person

ShcherbakovBGN/PCGN
ShcherbakovICAO 9303
ŜerbakovISO 9
Ščerbakovscholarly
ShherbakovGOST 7.79-B
Scherbakovad-hoc

Six Latin forms of one surname, five of them produced by a published standard. A matcher that romanises once picks a lane and mismatches every record written in another.

460labelled pairs

17baselines scored

7languages

100%reproducible from source data

Why fuzzy matching lies here

Fuzzy string matching assumes edit distance correlates with the probability of being a different entity. Across scripts, it does not — and the failure is not a tuning problem, because legitimate spelling variance and genuine identity difference sit at the same edit distance.

Shcherbakov ↔ Scherbakov

edit distance 1

Same person. Two romanisation systems disagreeing about щ.
Ivanov ↔ Ivanova

edit distance 1

Usually different people. Gendered forms of a surname shared by millions.
Aleksandr ↔ Oleksandr

edit distance 1

Genuinely ambiguous. Russian and Ukrainian forms of one given name; the records do not say which.

Layered on top: Kazakh exists simultaneously in Cyrillic, in competing Latin orthographies and in a Russian-mediated romanisation that collapses ә ұ ө ғ қ ң onto their nearest Russian letters. Patronymics appear, vanish and abbreviate. Turkic particles -ұлы / -қызы coexist with Russified -ович / -овна. Surnames are gendered, and a feminine ending stripped by one system is the same person while the gendered pair is two.

What was built

A labelled corpus of name pairs and a harness that scores any matcher against it. Hard negatives are the whole game: a corpus of easy negatives makes every matcher look excellent and teaches nothing.

Positives

Pairs known to be the same entity.

OFAC SDN alias sets: Cross-script and cross-romanisation variants curated by government analysts. Weak aliases excluded.
Wikidata labels: CC0. The ru/kk/uk and en labels of one item are the same entity by construction.
Rule-generated: Published transliteration tables, for rare-letter coverage. Capped at 30% of positives — synthetics only test what we already modelled.

Hard negatives

Different entities, string-similar on purpose. 72% of negatives.

Same surname: Different given name, distinct entities.
Gendered pairs: Иванов / Иванова, unrelated people.
Romanisation collisions: Surnames like Әбиев and Абиев that collapse to one Latin form — the sharpest negative available.
Shared patronymic: Same surname and patronymic, different person.

Discipline

The parts that decide whether it is honest.

Entity distinctness: Negatives require both dates of birth present and different — distinct identifiers alone are not proof.
Unresolvable pairs excluded: Identical names for different people are correctly labelled but unanswerable from names; including them would add a fixed error floor that measures nothing.
Conservative tagging: A romanisation standard is named only when it reproduces every token exactly. Under-attribution beats a confident wrong label.
Determinism: Byte-identical rebuild; every deterministic baseline reproduces exactly.

Corpus pilot-v0.1 · sha256 4eaae6355798c59d… · 460 pairs, 230 positive / 230 negative, 165 hard negatives, 68 synthetic positives · ru 243 · kk 66 · uk 61 · be 33 · ky 23 · tg 21 · uz 13

The result that matters

Recall per phenomenon for the strongest baseline, at an operating point holding the false-positive rate at 1% on hard negatives. The split is clean, and it is the same split for every baseline family tested.

Commodity tools already handle this

Pure romanisation variance

diacritics stripped		1.000
ICAO 9303		0.824
Uzbek Latin 1995		0.750
Uzbek apostrophe		0.750
BGN/PCGN		0.655

Nothing tested handles this

Name structure

patronymic abbreviated		0.000
patronymic Turkic `-ұлы`		0.000
patronymic dropped		0.083
token dropped		0.275
legal form `ТОО`/LLP		0.395

Recall at the 1% FPR operating point, hard-negative slice, best baseline (nomenklatura/logic-v2). Phenomena with fewer than 8 positives are omitted — recall over a handful of pairs is noise, not a finding.

Every baseline, hard-negative slice

Baseline	Family	max-F1	Recall @ 1% FPR	ROC-AUC	FPR to reach 95% recall
`nomenklatura/logic-v2`	OpenSanctions	0.8730	0.5435	0.8719	0.933
`embedding/LaBSE`	embedding	0.8120	0.3348	0.8161	0.927
`nomenklatura/name-based`	OpenSanctions	0.8008	0.3217	0.8236	0.855
`embedding/multilingual-e5-base`	embedding	0.7831	0.1261	0.7618	0.939
`icu-any-latin-ascii+levenshtein`	ICU translit	0.7811	0.4565	0.7978	0.836
`icu+soundex`	English phonetic	0.7452	0.0000	0.7359	1.000
`icu+double-metaphone`	English phonetic	0.7437	0.2217	0.6948	1.000
`nomenklatura/ofac`	OpenSanctions	0.7360	0.1826	0.5913	1.000
`jaro-winkler-raw`	naive fuzzy	0.7360	0.2043	0.6022	1.000
`exact-match-nfkc`	floor	0.7360	0.0000	0.5000	1.000

Selected rows; the full 17-baseline table is in the repository. Read max-F1 against 0.736, not zero — on this slice, calling every pair a match scores 0.736, which is why exact-match-nfkc ties several real matchers on F1 while its ROC-AUC of 0.5000 shows it never discriminates at all. F1 is a poor gate on a balanced corpus.

English phonetics fail, as predicted

icu+soundex reaches 0.0000 recall at 1% FPR — it has no high-precision operating point at all. Soundex and Metaphone encode English phonology; they collapse distinctions that matter in Slavic and Turkic names and preserve ones that don't. Any library defaulting to them for this data is mis-applied.

No transliteration, no chance

The ofac matcher scores non-zero on 1 of 186 cross-script positive pairs (highest score 0.333). It compares Latin tokens and does not romanise, so the cross-script case is invisible to it. Its ROC-AUC of 0.5913 is barely above chance.

Holding recall is ruinously expensive

To reach 95% recall, every baseline must accept a false-positive rate between 0.836 and 1.000 on hard negatives. The "loosen the threshold until it catches everything" behaviour is real and now measured — but note what that implies below.

The verdict, including what it disproves

IN-BETWEEN

The gate was written before the work, with numeric criteria: kill the idea if the best baseline reached 0.93 F1 and 0.95 recall at 1% FPR; proceed if it stayed at or below 0.85 F1 with errors clustering in nameable phenomena. The best baseline landed at 0.8730 F1 and 0.5435 recall at 1% FPR. Neither criterion fires.

Two things this measurement contradicts

The premise pointed at the wrong layer

The project was founded on the claim that modelling the transliteration transformation is the product. The measurement says that layer is substantially commoditised — the strongest baseline romanises well and recovers 1.000 of diacritic-stripped pairs and 0.824 of ICAO-romanised ones. What no tested matcher handles is name structure: patronymic conventions, Turkic particles, abbreviation, dropped tokens, legal forms. The gap is real, and it is one layer up from where it was expected.
The false-positive story did not reproduce

The commercial pitch was that incumbents flood compliance teams with false positives. At the 1% FPR operating point the strongest baseline produced one false positive across 115 string-similar hard negatives, and zero across gendered pairs, romanisation collisions, shared surnames and shared patronymics. It is precise. Its failure mode is missed matches — roughly 46% of true pairs — which is a regulatory exposure, not a labour cost. That is close to the opposite of the pitch.

What these numbers cannot tell you

Corpus difficulty is a choice, not a measurement. I built the negatives, so the headline recall is partly a property of that construction. Treat the hard slice as a lower bound on baseline quality, not a production estimate.
This is not a production false-positive rate. The corpus is roughly half positives; a real screening queue is overwhelmingly non-matches, so alert-volume figures from it would not transfer.
max-F1 is oracle-tuned. Thresholds are chosen knowing the labels, which flatters every baseline — deliberate, so the gate only fires against a baseline that fails at its best possible threshold.
Recall at 0.1% FPR is unmeasurable at this size and is reported as unresolvable rather than interpolated. It needs at least 1,000 negatives.
Positives skew canonical. Wikidata English labels tend to be the spelling an editor chose, often an English exonym, under-representing the ad-hoc variance that causes real pain.
An LLM-as-judge baseline has not been run — no API credential in the run environment. It is recorded as TBD, never estimated. The gate is not settled until it runs.
Thin outside the core. ky 23 pairs, tg 21, uz 13 — enough for a signal, not for per-language claims.

A pilot of 460 pairs, run 26 July 2026, against the OFAC SDN list published 24 July 2026. Sanctions data decays weekly; treat every figure as of those dates. The next steps are a full corpus of at least 5,000 pairs, the LLM baseline, and interviews that ask whether anyone would actually pay to fix patronymic handling — because the benchmark cannot answer that.

Reproducibility

Source snapshots are committed, so the corpus rebuilds offline and byte-identically. The report is generated from the metrics file rather than written by hand, so no figure in prose can drift from the measurement. Every baseline is open source or textbook — nothing behind a licence that forbids benchmarking, and no commercial product was evaluated.

Method, data and harness on GitHub ↗

Scope: defensive use only

A tamga is the seal Turkic and Mongol peoples stamped to mark identity and lineage — the oldest answer to "is this the same party?".

This work exists to help regulated parties find sanctioned counterparties. It will not become a tool for testing whether a name variant evades screening: no alias generators positioned around evasion, no "will this clear a watchlist?" checkers, no scoring surface that ranks variants by likelihood of not being flagged. Any matcher built on this benchmark is a screening aid; responsibility for compliance and for the disposition of every alert stays with the screener.