Skip to content

anonymization.md

Latest commit

 

History

History
881 lines (718 loc) · 43.1 KB

anonymization.md

File metadata and controls

881 lines (718 loc) · 43.1 KB
Raw

PII Anonymization

OpenMed's deidentify() API supports five redaction methods for detected PII entities:

Method Output Use when
mask [NAME], [EMAIL], … You want clear placeholders.
remove "" (deleted) You don't need positional alignment.
replace Locale-aware fake surrogates You need realistic-looking text.
hash NAME_a1b2c3d4 (entity-typed digest) You need consistent linking across docs.
shift_dates Dates only — shifted by N days You want to preserve relative time.

This document focuses on replace, which was upgraded in v1.3.0 to a full Faker-backed obfuscation engine. If you just want to compare all five methods side by side, start with the quickstart below.

Quickstart: choosing a method

mask — clear placeholders

from openmed import deidentify

result = deidentify(
    "Patient John Doe (DOB: 01/15/1970) called from 555-1234",
    method="mask",
)
print(result.deidentified_text)
# Patient [first_name] [last_name] (DOB: [date]) called from [phone_number]

Placeholder names come from the model's own entity labels, so they vary by model (the default OpenMed-PII-SuperClinical-Small-44M-v1 model used here splits names into first_name/last_name rather than a single NAME).

Not reversible by itself — pass keep_mapping=True and use reidentify() (see below) if you need to restore the original text later.

remove — delete PII entirely

result = deidentify("Call 555-1234", method="remove")
print(repr(result.deidentified_text))
# 'Call '

Use this when you don't need positional alignment with the original text (e.g. exporting de-identified text for search indexing).

replace — realistic fake surrogates

result = deidentify(
    "Email: test@example.com",
    method="replace",
    consistent=True,
    seed=42,
)
print(result.deidentified_text)
# Email: asnyder@example.com

Best for sharing data with downstream tools that expect well-formed values (e.g. an email field that should still look like an email). See The new replace engine below for locale and determinism options.

hash — consistent, irreversible digests

result = deidentify("Patient John Doe", method="hash")
print(result.deidentified_text)
# Patient first_name_a8cfcd74 last_name_fd53ef83

The same input always hashes to the same digest, so repeated mentions of the same value link together across documents — without storing the original anywhere.

shift_dates — preserve intervals, hide absolute dates

result = deidentify(
    "Admit 01/15/2020, follow-up 01/25/2020",
    method="shift_dates",
    date_shift_days=30,
)
print(result.deidentified_text)
# Admit 02/14/2020, follow-up 02/24/2020

The intent is for every date in a document to shift by the same offset, so durations between dates (e.g. "3 days after admission") stay correct. date_shift_days=30 is a fixed offset when no patient key is supplied.

For longitudinal research, pass a stable patient_key so every document for that patient receives the same HMAC-derived offset across sessions:

patient_token = load_patient_key_from_vault()
hmac_key_material = load_date_shift_hmac_key()

shared_kwargs = {
    "method": "shift_dates",
    "patient_key": patient_token,
    "date_shift_max_days": 365,
    "date_shift_secret": hmac_key_material,
}

first = deidentify("Visit 01/15/2020", **shared_kwargs)
second = deidentify("Visit 03/15/2020", **shared_kwargs)

Equal patient keys and the same secret yield identical offsets, preserving intervals across documents. Different patient keys generally produce different offsets within date_shift_max_days. The raw patient key is used only as HMAC input and is not returned in shifted text, mappings, logs, or audit artifacts. Patient-keyed offsets require caller-supplied date_shift_secret; do not use PHI as that key material. If patient_key is supplied with the older date_shift_days option, that value is treated as the maximum absolute offset bound; prefer date_shift_max_days for new code.

Solar Hijri, Ethiopian and Thai Buddhist-era dates

shift_dates shifts Solar Hijri (Jalali), Ethiopian and Thai Buddhist-era dates inside their own calendar. Each date is converted to an absolute day number, moved by the same per-patient offset as the document's Gregorian dates, and rendered back in the source calendar, digit system, separators, zero padding, era marker and month-name form:

Pack Source span date_shift_days=1
fa ۱۴۰۴/۰۷/۳۰ ۱۴۰۴/۰۸/۰۱
fa ۳۰ اردیبهشت ۱۴۰۴ ۳۱ اردیبهشت ۱۴۰۴
th 28 กุมภาพันธ์ 2567 29 กุมภาพันธ์ 2567
am 05/13/2016 (5 Pagume 2016) 01/01/2017
am ጳጉሜ 5 ቀን 2016 ዓ.ም መስከረም 1 ቀን 2017 ዓ.ም

The calendar of a date span is chosen in this order:

  1. An era marker inside the span, such as ه.ش or هجری شمسی (Solar Hijri), พ.ศ. or B.E. (Buddhist era), ዓ.ም or E.C. (Ethiopian), or a Gregorian marker such as میلادی, م, ค.ศ., እ.ኤ.አ or G.C.. A detector or custom recognizer can instead attach a calendar code to the span with metadata={"calendar": "solar_hijri"}.
  2. Month names that belong to one calendar: the Persian Solar Hijri months and the Amharic Ethiopian months, including Pagume.
  3. The language pack's default_calendar (fa: solar_hijri, am: ethiopian, th: thai_buddhist, ar and ur: hijri; every other pack: gregorian) when the year falls inside that calendar's supported range. Otherwise a year from 1800 to 2299 is treated as Gregorian and shifted exactly as before.

The conversions are exact standard-library arithmetic over these ranges:

Calendar Code Years Gregorian span Rule
Solar Hijri solar_hijri 1277–1501 Mar 1898 – Mar 2123 33-year leap cycle, where it matches astronomical year lengths
Ethiopian ethiopian 1891–2114 Sep 1898 – Sep 2122 Pagume has 6 days when year % 4 == 3, otherwise 5
Thai Buddhist era thai_buddhist 2484–2665 1941 – 2122 B.E. = C.E. + 543 since 1941, when the year began on 1 January

A native-calendar date keeps the [DATE_SHIFTED] placeholder instead of being guessed when it does not exist in its calendar (۳۰ اسفند ۱۴۰۴, Pagume 6 of a common year), when its year or the shifted result is outside the supported range, when the year has only two digits, when the day/month order or the era markers conflict, or when the calendar or variant has no converter (a span tagged {"calendar": "hijri_diyanet"} stays unshifted). The returned reason codes never contain the date.

Numeric dates in Amharic text are read as Ethiopian dates. Mark Gregorian dates with እ.ኤ.አ or G.C., or attach {"calendar": "gregorian"} from detection, to shift them in the Gregorian calendar. keep_year=True restores the source year in the source calendar and clamps Esfand 30 and Pagume 6 the same way 29 February is clamped.

replace and format_preserve generate surrogates for these spans in the same calendar and format, drawn from the 1925–2024 window used for Gregorian date surrogates, so a Jalali source receives a year between 1303 and 1403.

To add a calendar, subclass openmed.core.calendars.CalendarConverter (conversion to and from the shared datetime.date.toordinal() day number, month lengths, and the supported year range), register it once with register_calendar(...), and reference its code from a pack's default_calendar or a span's calendar metadata.

Lunar Hijri and Bikram Sambat dates

Lunar Hijri and Bikram Sambat months cannot be computed with arithmetic: Hijri month starts follow a declared variant, and Bikram Sambat months last 29 to 32 days according to published tables. OpenMed ships two bounded, versioned month-length tables and shifts these dates through them with the same per-patient offset as the other calendars:

Code Calendar Years Gregorian span
hijri Lunar Hijri, Umm al-Qura (Saudi civil) 1356–1450 AH 14 Mar 1937 – 13 May 2029
bikram_sambat Bikram Sambat as published in Nepal 2000–2083 BS 14 Apr 1943 – 13 Apr 2027
Pack or span metadata Source span date_shift_days=3
ar (Safar 1447 has 29 days) 28/02/1447 02/03/1447
ar ١٥ رمضان ١٤٤٧ هـ ١٨ رمضان ١٤٤٧ هـ
{"calendar": "bikram_sambat"} 2082/03/30 2082/04/01
any (Asar 2082 has 32 days) २०८२ असार ३२ गते २०८२ साउन ३ गते

These dates are recognized from era markers (هـ, ه, هجري, ه.ق, قمری, ھ, A.H. for Hijri; वि.सं., विक्रम संवत्, B.S. for Bikram Sambat; ई.सं. marks a Gregorian date in Nepali text), from the {"calendar": "hijri"} and {"calendar": "bikram_sambat"} codes that month-name detection attaches, and from Hijri month names in Arabic, Persian and Urdu spelling or Bikram Sambat month names in colloquial and formal Nepali spelling, written day-month-year, month-day-year or year-month-day. When a shift changes the month, the new month is written in the source's convention (ذی‌القعده in Persian, ذوالقعدہ in Urdu, भाद्र after श्रावण). The ar and ur packs declare hijri, so an unmarked numeric date whose year is in the table is read as an Umm al-Qura date; the Gregorian years 1800–2299 never overlap the table and keep shifting as before. No bundled pack declares bikram_sambat, so unmarked numeric dates in Nepali text stay Gregorian unless they are marked or tagged.

The placeholder is kept when the year or the shifted result is outside the table, when the day does not exist in the table (30 Safar 1447, 32 Shrawan 2082), when markers or codes conflict, and when a span declares a variant without a registered table, such as {"calendar": "hijri_diyanet"}. replace and format_preserve surrogates stay in the table's calendar, between 1356 and 1446 AH or 2000 and 2081 BS.

Each table in openmed/core/data/calendars/ records its format and table version, SPDX license, copyright and license text, source project, version and retrieval date, a verification record, and a SHA-256 digest of its canonical JSON. The loader recomputes the digest, requires a permissive license, and checks the bundled tables against digests pinned in openmed.core.calendar_tables; a failing table is refused with a ConfigurationError that names only the failing field.

  • hijri_umm_al_qura.json keeps the 1356–1450 AH month starts of hijridate 2.6.0 (MIT), which tabulates the archived Umm al-Qura almanac. Every month start matches R.H. van Gent's Umm al-Qura table except 1427-06 and 1446-06, where ICU 77.1 islamic-umalqura matches the bundled value, and ICU matches every month from 1420 AH. Earlier years have no second tabulation and later years are projections on which the references diverge. Shaban 1364 AH has 28 days in both almanac tabulations and is kept.
  • bikram_sambat.json keeps the 2000–2083 BS month lengths of py-nepali 1.2.0 (MIT). nepali-datetime 1.0.8.5 and pyBSDate 0.3.2 (both MIT) match every year except 2062 and 2000 respectively, where the other two tables agree. Earlier years disagree across the tables and from 2084 BS their projections diverge.

Load a table for another variant, or a longer range, from your own file. A table registered as hijri replaces the Umm al-Qura table for every Hijri date; a table registered under a new code, such as hijri_pk, applies only to spans whose calendar metadata names that code:

from openmed.core.calendar_tables import load_calendar_table
from openmed.core.calendars import register_calendar

converter = load_calendar_table("hijri_local.json", expected_sha256="<sha256>")
register_calendar(converter, replace=True)

A caller-supplied table under a non-permissive license loads only with allow_non_permissive_license=True and must never be bundled with OpenMed. calendar_table_digest(document) computes the sha256 field for a table you write.

Reversing a de-identification: reidentify()

Pass keep_mapping=True to get back a mapping you can hand to reidentify() later:

from openmed import deidentify, reidentify

text = "Dr. Alice Smith met Bob Jones today"
result = deidentify(text, method="mask", keep_mapping=True)
print(result.deidentified_text)
# Dr. [first_name] [last_name] met [first_name_2] [last_name_2] today

restored = reidentify(result.deidentified_text, result.mapping)
assert restored == text

Repeated entities of the same type (two first_names above) get a numbered placeholder ([first_name], [first_name_2], ...) so each one maps back to its own original value — this was a known limitation (#204) fixed by #222; reidentify() now round-trips correctly even when a type repeats.

Custom deny-list and allow-list recognizer

Use custom_recognizer when your site has identifiers the model does not know, or benign values that must never be redacted. The argument accepts a plain mapping, a CustomRecognizer instance, or a .json/.yaml path.

from openmed import deidentify, extract_pii

custom_recognizer = {
    "case_sensitive": False,
    "deny": {
        "terms": [
            {"term": "Ward Phoenix", "label": "LOCATION"},
        ],
        "patterns": [
            {"pattern": r"\bSTUDY-\d+\b", "label": "ID_NUM"},
        ],
    },
    "allow": {
        "terms": ["Mercy Trial"],
        "patterns": [r"\bPUBLIC-\d+\b"],
    },
}

entities = extract_pii(text, custom_recognizer=custom_recognizer)
result = deidentify(text, method="mask", custom_recognizer=custom_recognizer)

Deny-list terms are literal strings. Deny-list patterns are regular expressions. Each deny entry needs a label; OpenMed keeps that label on the returned entity and normalizes it into the canonical label taxonomy for policy and audit handling. Matches are emitted with custom:deny provenance.

Allow-list terms and patterns suppress any overlapping span from any detector, including model detections, deterministic rules, and custom deny-list matches. Allow-list precedence always wins over deny-list and model detections, so an allowed value is left untouched in deidentify() output.

Recognizer metadata stores hashes and rule ids, not raw matched surfaces. In the staged pipeline, custom matching runs on normalized text and spans are remapped back to original offsets before redaction.

The new replace engine

method="replace" no longer picks from a small hardcoded list. It builds a per-document Anonymizer (see openmed.core.anonymizer) that delegates to Faker with custom providers for clinical IDs.

from openmed import deidentify

deidentify(
    "Paciente Pedro Almeida, CPF: 123.456.789-09",
    method="replace",
    lang="pt",
    locale="pt_BR",          # default for pt is pt_PT; override per call
    consistent=True,         # same input -> same surrogate within doc
    seed=42,                 # cross-run reproducibility
)

Locale resolution

lang (an ISO 640-1 code OpenMed uses everywhere) maps to a Faker locale via LANG_TO_LOCALE:

OpenMed lang Faker locale Notes
en en_US
fr fr_FR
de de_DE
it it_IT
es es_ES
nl nl_NL
hi hi_IN
te en_IN Faker has no Telugu locale — emits a one-time warning.
pt pt_PT Override with locale="pt_BR" for Brazilian Portuguese.

Pass locale= explicitly to override per call (e.g. pt_BR to generate CPF/CNPJ surrogates instead of Portuguese NIF/VAT).

For Mexican Spanish records, pass locale="es_MX". The deterministic safety sweep recognizes context-labelled CURP and RFC values, and replacement uses checksum-valid synthetic surrogates while preserving whether an RFC is the 12-character company form or 13-character individual form.

Country-aware African French and Portuguese surrogates are available through conceptual locale overrides. fr_SN, fr_CI, and fr_CM use curated names, cities, addresses, and country-code phone formats while keeping the French PII model; pt_MZ and pt_AO do the same with the Portuguese model. Unsupported Faker methods delegate to fr_FR or pt_PT, so these overrides do not alter the default fr and pt output. Arabic also accepts ar-DZ and ar-MA; if the installed Faker release lacks the requested regional backend, OpenMed falls back to ar_EG and emits the same one-time warning used by the other Arabic regional overrides.

For the full per-language table — every SUPPORTED_LANGUAGES code with its default PII model, Faker locale, and a before/after example — see Per-Language De-identification.

Devanagari and Telugu name surrogates

Hindi and Telugu names written in their own script are replaced with pronounceable names built from aksara, the written syllables of both scripts. This covers the PERSON, FIRST_NAME, MIDDLE_NAME, and LAST_NAME labels. Each aksara is a consonant or a virama conjunct, an optional dependent vowel sign, and an occasional anusvara or visarga. A name may also start with an independent vowel. The consonants, vowel signs, and conjuncts come from small inventories in openmed.core.anonymizer.providers.aksara_names. OpenMed bundles no list of real names for these scripts.

Every surrogate:

  • has the same number of space-separated tokens as the source, in the same order. An initial such as ए. stays a one-aksara initial, and hyphens are kept;
  • gives each token the source token's aksara count, between 2 and 4 for Devanagari or 2 and 5 for Telugu;
  • reuses none of the source's code points in that script: consonants, vowels, vowel signs, virama, anusvara, or visarga. A surrogate therefore never equals its source;
  • splits, under the grapheme rules in openmed.core.decoding.spans, into aksara that are each well formed; and
  • is reproducible for a fixed seed.
from openmed.core.anonymizer import Anonymizer
from openmed.core.anonymizer.providers.aksara_names import (
    is_well_formed_aksara_name,
)

anonymizer = Anonymizer(lang="hi", consistent=True, seed=11)
surrogate = anonymizer.surrogate("अनीता शर्मा", "PERSON")
# -> "पजूग वके": two tokens, no consonant shared with the source
assert is_well_formed_aksara_name(surrogate, script="Devanagari")

Latin-script names in Hindi or Telugu notes still use Faker. Marathi keeps its own generator, which handles honorifics and three-part names. Other language packs that use Devanagari can register generate_devanagari_aksara_name with register_label_generator(..., language_pack=pack, script="Devanagari").

Determinism

Three modes:

  • Random (default). Every call samples fresh surrogates. Good for audits or when you want visible variability.
  • consistent=True. Same (canonical_label, original_value) pair resolves to the same surrogate within the call. "John Doe" appearing twice in one document gets one surrogate.
  • seed=<int> (implies consistent=True). Same seed across runs produces the same surrogate stream — useful for snapshot tests and regression fixtures.

Determinism uses hashlib.blake2b over (seed, canonical_label, original), so different originals always get different surrogates.

Cross-document surrogate vaults

Canonical span and pipeline audit-record HMACs use private random keys by default. A Pipeline keeps its key for its own lifetime, so streaming windows and repeated calls on that instance agree. Separate pipelines, standalone recognizer calls, and interop projections receive separate keys. For intentional stable HMACs across pipeline instances, pass the same non-empty, securely stored hmac_secret to each Pipeline. Fresh deidentify(..., audit=True) calls therefore produce different audit hashes; each report still verifies against its own contents. This does not change redaction labels or offsets.

The separate AuditReport format retains SHA-256 integrity digests of input, spans, and context. These unkeyed digests can be guessed for low-entropy values; they are not an anonymization boundary. Keep those reports access-controlled.

The web SDK and React Native bridges generate a private key per inference call when hashSecret is omitted. Supply a non-empty private hashSecret when cross-call linkage is needed; keep keys separate from span output.

Use a SurrogateVault when separate deidentify(..., method="replace") calls need stable pseudonyms for the same identifier:

from openmed import SurrogateVault, deidentify

vault = SurrogateVault.from_file(
    "surrogate-vault.json",
    hmac_secret="rotate-and-store-this-secret-outside-the-vault",
)

first = deidentify(
    "Patient John Doe was admitted.",
    method="replace",
    surrogate_vault=vault,
)
second = deidentify(
    "John Doe returned for follow-up.",
    method="replace",
    surrogate_vault=vault,
)

The vault file stores (canonical_label, lang, HMAC text_hash) -> surrogate entries plus schema_version and hmac_scheme; it does not store raw source surfaces or the HMAC secret. Treat the file as sensitive pseudonymous linkage data anyway: it can connect records across documents even without plaintext.

For person names, the vault derives that HMAC input from a deterministic ISO 15919 pivot. Devanagari, Bengali, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada, Malayalam, and ISO 15919 inputs can therefore share one in-memory join key and one surrogate without storing the romanized name. The same engine can normalize explicit ITRANS and Harvard-Kyoto input before conversion:

from openmed.processing import from_latin, to_latin, transliteration_key

result = to_latin("राम ராம rāma")
assert result.text == "rāma rāma rāma"
assert result.remap_span(0, 4) == (0, 3)
assert transliteration_key("राम") == transliteration_key("ராம")
assert transliteration_key("rAma", "ITRANS") == transliteration_key("rāma")
assert from_latin("lakShmI", "Devanagari", scheme="itrans") == "लक्ष्मी"

The default schwa_policy="preserve" and anusvara_policy="marker" form the round-trip-safe subset. schwa_policy="source" applies northern word-final schwa deletion, while anusvara_policy="homorganic" expands a nasal according to the following consonant; both are intentionally lossy. The public LOSSY_CASES tuple also lists nukta and extended letters, Tamil distinctions that its orthography does not encode, Gurmukhi addak, and Malayalam chillu letters. The stdlib-only tables are a clean-room Unicode/ISO implementation interoperable with the conventions of Aksharamukha (AGPL-3.0) and the Indic NLP Library (MIT). No code, data, copyleft component, neural weights, or third-party mapping bundle from either project is included.

Perso-Arabic Urdu has a built-in ur_IN language pack for script routing, deterministic patterns, and Indian identifiers. It is not covered by this ISO 15919 transliteration engine; Urdu transliteration still fails closed with ValueError and requires an out-of-process adapter.

Format preservation

Phone numbers, dates, and emails preserve the structure of the original:

deidentify("Call (415) 555-1234", method="replace", consistent=True, seed=1)
#  -> "Call (XXX) XXX-XXXX"  (digit groups, separators, country code position)

deidentify("Born 01/15/1970", method="replace", consistent=True, seed=1)
#  -> "Born MM/DD/YYYY"      (separator and ordering kept)

deidentify("Email: john@hospital.org", method="replace", lang="en")
#  -> "Email: alice@hospital.org"  (domain kept, local part faked)

Person names: honorifics, gender cues and name order

PERSON surrogates keep the shape of the source name instead of dropping its title or swapping in a name that contradicts the surrounding grammar:

  • No inserted titles. Faker's name formats randomly add titles and post-nominals (Bi., Chị, Univ.Prof., جناب آقای, MBA.). A surrogate never carries a title the source name lacked, in any language. Surname particles such as de, du or Thai ณ <place> are kept.
  • Source honorifics are kept verbatim. For Thai, Amharic, Swahili, Arabic, Persian, Urdu, Hindi, Vietnamese, Indonesian, Yoruba and Hausa a registered honorific prefix or suffix is peeled off, only the name is replaced, and the honorific is re-attached with its original spacing.
  • Gender comes only from the span itself. When that honorific (or a Vietnamese Thị/Văn middle name) marks gender, the surrogate's given name has the same gender. OpenMed never infers gender from a name.
  • Order follows the language. Vietnamese surrogates are family name first.
  • An honorific-only span is masked ("Bi." becomes [PERSON]), never replaced by a bare title.
from openmed.core.anonymizer import Anonymizer

Anonymizer(lang="th", consistent=True, seed=0).surrogate(
    "นางสาวสมหญิง ใจดี", "PERSON"
)
#  -> "นางสาวธัชญา ศรีวงค์"  (title kept, female given name)

Anonymizer(lang="vi", consistent=True, seed=1).surrogate("Lê Thu Hà", "PERSON")
#  -> "Nguyễn Mai"  (family name first, no title added)

Words that also work as names stay part of the name: a title is recognized only as a whole token at the start or end of the span (Thai titles, which are written attached, must be followed by a valid syllable start), so Urdu سید/بیگم, a trailing Arabic السيد or Hindi श्रीकांत are replaced like any other name. Japanese, Korean and Chinese keep their own honorific handling in openmed.core.name_order. Add a title for another language with openmed.core.person_honorifics.register_person_honorific(lang, text, gender="female").

Native digit systems

replace, format_preserve and shift_dates render their output in the digit system of the source span. A Persian national ID such as ۰۰۱۲۳۴۵۶۷۹ gets a surrogate in Persian digits, an Arabic-Indic date ١٥/٠٣/٢٠٢٦ shifted by three days becomes ١٨/٠٣/٢٠٢٦, and a grouped Devanagari Aadhaar keeps its digits and its #### #### #### layout. ASCII surrogates in a native-digit note would otherwise mark every replaced span.

Generators and validators still work on ASCII digits. OpenMed folds the source span to ASCII, runs the generator or the date shifter on the folded value, and then renders the digits of the result back. The folded surrogate therefore passes the same checksum and structure validators as an ASCII surrogate. In deterministic mode the same identifier written in different digit systems gets the same surrogate digits. The helpers live in openmed.core.anonymizer.digit_systems.

The rules are:

  • Any Unicode decimal digit system works, including Arabic-Indic, Extended Arabic-Indic (Persian and Urdu), Devanagari, Bengali, Thai, Myanmar and fullwidth digits.
  • Spans whose digits are all ASCII, or that contain no digits, are unchanged.
  • A mixed-digit span uses its dominant system, the one with the most digits. A tie goes to the system whose first digit comes earliest. The whole surrogate uses that one system.
  • The Arabic date separator ؍ and the fullwidth ／, － and ． are kept in shifted and generated dates. If the span also contains the matching ASCII separator, the native separator is left as it is.
  • If a generator returns bare digits for a grouped source with the same digit count, the source grouping and separators are applied to the surrogate.
  • Ethiopic numerals (U+1369 to U+137C) are not positional decimal digits and have no zero, so they are never converted. Spans that use them behave as before.
  • Only digits and separators are rendered here. Calendar handling for Jalali, Buddhist-era and other non-Gregorian dates is a separate step.

The mapping and audit report hold the same values as for ASCII input: the mapping links each rendered surrogate to its source span, and the audit report stores no extra copy of the source span or of its ASCII form.

Clinical ID checksums

OpenMed reuses the existing checksum validators in openmed.core.pii_i18n so every surrogate ID passes the same validator that detection uses:

Locale ID type Provider
pt_BR CPF Faker built-in (pt_BR.cpf)
pt_BR CNPJ Faker built-in (pt_BR.cnpj)
nl_NL BSN (Elfproef) Faker built-in (nl_NL.ssn)
fr_FR NIR Faker built-in (fr_FR.ssn)
it_IT Codice Fiscale Faker built-in (it_IT.ssn)
es_ES NIE Faker built-in (es_ES.nie)
es_MX CURP OpenMed MexicanCURPProvider (modulo-10)
es_MX RFC OpenMed MexicanRFCProvider (modulo-11)
en_IN Aadhaar (Verhoeff) OpenMed AadhaarProvider (Faker's built-in is invalid)
de_DE Steuer-ID OpenMed GermanSteuerIdProvider (Faker's de_DE.ssn is US-style)
any NPI (Luhn over 80840) OpenMed NPIProvider
any Generic MRN OpenMed MedicalRecordNumberProvider

Extending

from openmed import register_clinical_provider, register_label_generator
from faker.providers import BaseProvider

# Add your own checksum-bearing ID format
class HospitalAccountProvider(BaseProvider):
    def hospital_account(self):
        return f"HACC-{self.numerify('########')}"

register_clinical_provider(HospitalAccountProvider)

# Override the generator for a canonical label
def my_first_name(faker, original, *, locale):
    return faker.first_name() + "-test"

register_label_generator("FIRST_NAME", my_first_name)

Live surrogate quality

openmed.eval.surrogate_quality grades committed fixture records. openmed.eval.surrogate_quality_live grades the surrogates the registered providers generate today. It reads the language-pack registry when it runs, so a pack registered later is scored without code changes. National-ID-only languages are scored as well.

For each language it builds seeded synthetic originals in the language's expected script and generates surrogates for six labels: name, address, city, phone, id and date. Each sample is scored on the fixture scorer's dimensions:

Dimension Live check
script Letters in names, addresses and cities fall in the scripts the pack declares.
format Dates follow the locale date order. Addresses and cities do not reuse the US Faker template. Phones outside NANP regions do not take a NANP shape.
checksum National-ID surrogates pass the language's registered validators.
consistency A fresh provider with the same seed reproduces the value, which neither equals nor retains the original.
naturalness No fallback placeholders, digits in names or cities, inserted honorifics, or implausible name shapes.

Script expectations come from each pack's declared scripts. Packs routed without a script claim (id, ro) and national-ID-only languages use LIVE_SURROGATE_SCRIPT_FALLBACKS. A cell that cannot be scored is reported as unscored with a reason code such as no_script_expectation or no_national_id_provider instead of being skipped.

from openmed.eval.release_gates import evaluate_live_surrogate_quality_gate
from openmed.eval.surrogate_quality_live import evaluate_live_surrogate_quality

report = evaluate_live_surrogate_quality()  # every registered pack, offline
report.matrix()["fr"]    # {"name": "pass", "address": "pass", ...}
report.failing_cells     # ("<language>/<label>", ...) below the floor
report.unscored_cells    # cells the live scorer could not grade

gate = evaluate_live_surrogate_quality_gate(report)
gate.details["failing_cells"]  # rates and reason counts per failing cell

Each cell passes when at least 90% of its seeded samples pass every applicable dimension. The gate fails closed: it passes only when every language and label is covered and every cell passes or carries an exemption. A cell can be exempted only for a locale with a documented surrogate-locale approximation (a pack's surrogate_locale_approximation, or a documented locale approximation such as Telugu's en_IN backend), and the exemption must record a reason:

report = evaluate_live_surrogate_quality(
    exemptions=[
        {
            "language": "te",
            "labels": ["address", "city"],
            "reason": "Telugu generic fields use the en_IN backend",
        }
    ]
)

Reports and gate details contain status codes, counts, rates and truncated SHA-256 digests of generated values. They never contain an original or a surrogate value. To run the gate from the command line:

python -m openmed.eval.release_gates --live-surrogate-quality \
  --output live-surrogate-quality.json

The command exits with 1 when any cell fails or is unscored. A release benchmark report opts in with surrogate_quality_live: true in its metrics or metadata, and surrogate_quality_live_required: true makes that evidence mandatory. Release evidence may add exemptions, but it can only raise the pass-rate floor and the sample count, and the seed is fixed.

Chinese word segmentation

Chinese routing uses a pluggable word segmenter that preserves exact Python string offsets. The default jieba backend and the small synthetic clinical dictionary ship with OpenMed; no model download is required:

from openmed.core.config import OpenMedConfig
from openmed.processing import create_chinese_segmenter_from_config

config = OpenMedConfig(
    chinese_segmentation_backend="jieba",
    chinese_user_dict_path="/srv/openmed/zh_terms.txt",
)
segmenter = create_chinese_segmenter_from_config(config)
tokens = segmenter.segment("患者王芳因心房颤动入院")

assert all(token.text == "患者王芳因心房颤动入院"[token.start:token.end]
           for token in tokens)

An additional dictionary is a UTF-8 text file with one entry per line in term, term frequency, or term frequency POS form. For example:

心脏超声 90000 nz
临床路径

Keep organization-specific dictionaries outside the package and point chinese_user_dict_path or OPENMED_CHINESE_USER_DICT at the local file. This avoids bundling private terminology or restricted lexicons.

The optional pkuseg (MIT) and HanLP (Apache-2.0) adapters are installed with openmed[zh-pkuseg] and openmed[zh-hanlp], respectively. Set chinese_segmentation_backend or OPENMED_CHINESE_SEGMENTATION_BACKEND to select one. pkuseg defaults to the medicine domain, configurable through chinese_pkuseg_domain; HanLP accepts a preloaded tokenizer or a local model path through create_chinese_segmenter(..., hanlp_model=...). OpenMed never downloads optional model files implicitly, so provision those assets before selecting either backend.

For the cache and storage layout of locally provisioned pkuseg and HanLP assets, deployment and upgrade/rollback procedures, institutional dictionary governance, and segmentation regression checks, see Chinese Segmentation Operations.

Privacy-filter family

OpenMed ships three privacy-filter families, all the same OpenAI Privacy Filter architecture (gpt-oss-style sparse-MoE transformer with local attention, sink tokens, RoPE+YaRN, tiktoken o200k_base), differing only in their training data:

The per-language PII API uses openmed.core.pii_i18n.SUPPORTED_LANGUAGES as its source of truth and supports 42 supported PII language codes: am, ar, as, bn, cs, da, de, el, en, es, fa, fr, gu, he, hi, id, it, ja, kn, ko, ml, mr, nl, no, or, pa, pl, pt, ro, ru, sv, sw, ta, te, th, tr, uk, ur, vi, xh, zh, and zu. Russian routing currently uses a documented multilingual default-model placeholder. Bengali, Chinese, and Tamil have dedicated registry entries. The optional Indic NER adapter can serve Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu. It loads only an explicit path or repository from OPENMED_INDIC_NER_MODEL and has no bundled default checkpoint. See the Indic NER checkpoint compatibility matrix for supported label maps, offset contracts, and opt-in real-checkpoint tests. Additional validator-backed national-ID providers cover ID-only locales such as Latvian, Slovak, Malay, Filipino, and Finnish without adding default PII models for those language codes. Urdu defaults to ur_IN with an installed en_IN Faker backend for general surrogate data and an Aadhaar provider. An explicit ur_PK locale retains CNIC validation. The Irish en_IE overlay validates PPS numbers with their weighted modulo-23 check letter, and the Japanese ja_JP path validates My Numbers with their modulo-11 check digit; both paths generate synthetic, checksum-valid surrogates locally. The multilingual privacy-filter family is a checkpoint family; it does not expand the per-language API allow-list.

Variant Trained on PyTorch artifact MLX (full) MLX (8-bit)
OpenAI Privacy Filter OpenAI's PII training set openai/privacy-filter OpenMed/privacy-filter-mlx OpenMed/privacy-filter-mlx-8bit
OpenAI Nemotron Privacy Filter Nemotron PII dataset OpenMed/privacy-filter-nemotron OpenMed/privacy-filter-nemotron-mlx OpenMed/privacy-filter-nemotron-mlx-8bit
OpenMed Multilingual Privacy Filter OpenMed multilingual PII corpus; shared fallback for multilingual routes, including the documented ru placeholder OpenMed/privacy-filter-multilingual OpenMed/privacy-filter-multilingual-mlx OpenMed/privacy-filter-multilingual-mlx-8bit

All run through the same extract_pii() / deidentify() API — only the weights differ:

extract_pii(text, model_name="OpenMed/privacy-filter-mlx-8bit")
extract_pii(text, model_name="OpenMed/privacy-filter-nemotron-mlx-8bit")
extract_pii(text, model_name="OpenMed/privacy-filter-multilingual-mlx-8bit")

deidentify(text, model_name="OpenMed/privacy-filter-nemotron",
           method="replace", consistent=True, seed=42)
deidentify(text, model_name="OpenMed/privacy-filter-multilingual",
           method="replace", consistent=True, seed=42)

Backend selection. On Apple Silicon with MLX importable, the MLX artifact runs natively via PrivacyFilterMLXPipeline. Elsewhere, the call substitutes the corresponding PyTorch model via transformers and emits a one-time UserWarning explaining the swap. The fallback is family-aware — an MLX-only Nemotron request on Linux substitutes OpenMed/privacy-filter-nemotron, and an MLX-only multilingual request substitutes OpenMed/privacy-filter-multilingual, so the user gets the same training distribution they asked for.

Either way the output entity dicts have the same shape so the rest of the pipeline behaves identically. Smart-merging (regex-based span construction) is skipped for this family — the model already does Viterbi-constrained BIOES decoding internally.

The dispatch lives in openmed.core.backends.create_privacy_filter_pipeline. To register a new fine-tune that should fall back to its own PyTorch repo on non-Mac hosts, add a row to _TORCH_FALLBACK_BY_FAMILY in that module. If a fine-tune introduces a genuinely different architecture (not just new weights), it would also need a new MLX model class and family branch in openmed.mlx.models.build_model — but a same-architecture fine-tune needs neither.

Backwards compatibility

replace outputs are no longer drawn from the prior hardcoded list (["Jane Smith", "John Doe", "Alex Johnson", "Sam Taylor"], etc.). Any test that asserted on those exact strings must either:

  • pass consistent=True, seed=<value> and update the expected output, or
  • assert non-equality with the original instead of equality with a hardcoded surrogate.

Other methods (mask, remove, hash, shift_dates) are unchanged.