[rwagwani]

One PR in place of #193, #186 and #197, as asked. Those three were closed on 2026-08-13 until CI was fixed. CI on main is green now, and all three problems still reproduce on main at 9214f43.

The three PRs were built on b28c027 (2026-08-13). This branch is rebased onto 9214f43 (2026-10-23). None of the 69 commits in between touch the blocklist, so the rebase applied without conflicts. Every number below was measured on the new base.

Every open review point from those PRs is addressed (see the table at the end). Two more gaps of the same kind turned up while measuring, so this PR closes them too: Cyrillic and Greek lookalike letters, and an invisible character inside one word of a multi-word entry.

What this PR combines

The commits from the three closed PRs are kept as they were reviewed and rebased without changes, so each one can still be read on its own. The commits marked "new" were written for this PR.

Commit	From	What it does
a2375d8	#193	Folds fullwidth and accented letters before to_ascii deletes them
a5b8d95	#186	Lets the whitelist hold phrases such as desk in
a6fa0f7	#186	hygiene.py: flags entries that two ordinary words fuse into
f0cb4f9	#186	Makes the hygiene probe check what the gate advises
c7235a4	#186	Warns at load time when a whitelisted word is still censored as a leetspeak variant of another entry
a47282d	#186	Exempts only the matches inside a phrase, not the phrase itself
0517c57	#197	coverage.py: measures what a list change stops catching
bc01aef	new	Makes the raw pass additive; one list-loading path; keeps the leetspeak characters in phrase matching; the audit uses the production reader
c5c2a72	new	Phrase tests go through is_safe; adds zero-width and phrase-boundary tests
fd9fdbd	new	Folds Cyrillic and Greek letters that are drawn like Latin ones
5119bef	new	Also reads invisible characters as deleted, not only as a space

#193, #186 and #197 stay closed. #201 (reasoner video decode) is not included: it is unrelated code and is on hold separately.

The problems, on today's main

All of these were checked with the shipped Cosmos-Guardrail1 lists and the real WordNet lemmatizer, through is_safe.

Prompt	main	This PR
a ｓｎｏｗ white poster (fullwidth)	passes	blocked
a snöw white poster (accent)	passes	blocked
a sn<U+043E>w white poster (Cyrillic o)	passes	blocked
a sn<U+200B>ow white poster (zero-width inside a word)	passes	blocked
a boston·dynamics robot (middle dot)	blocked	blocked
a robot sitting at a desk in the background	blocked	blocked by default; passes once desk in is whitelisted
a robot at a desk<U+200B>in the back (zero-width), with desk in whitelisted	n/a	blocked
a camera shot gun in frame, with camera shot whitelisted	n/a	blocked

1. Fullwidth and accented spellings get past is_safe (was fix(guardrail): fold code points before to_ascii erases them #193). is_safe runs to_ascii first, which deletes non-ASCII characters instead of folding them. So the folding added in fix(guardrail): whitelist restore corrupts the censored prompt, and invisible characters evade the blocklist #183 never sees them.
2. Lookalike letters get past is_safe (new). A Cyrillic or Greek letter drawn like a Latin one survives NFKC, and to_ascii deletes it. On the shipped list, 381 of 383 entries pass when spelled this way.
3. An invisible character inside one word of a multi-word entry hides it (new). fix(guardrail): whitelist restore corrupts the censored prompt, and invisible characters evade the blocklist #183 reads an invisible character as a space, which catches snow<U+200B>white. The same rule turns sn<U+200B>ow white into sn ow white, which matches nothing. 44 entries are affected.
4. Ordinary prose fuses into an entry, and the whitelist cannot fix it (was fix(guardrail): let the whitelist hold phrases, so ordinary prose stops matching blocklist entries #186). The matcher's whitelist only holds single tokens, so a phrase like desk in never reaches it.
5. There is no way to measure what an unblocking change gives up (was feat(guardrail): measure what a blocklist change stops catching #197). A corpus of ordinary output shows new blocks, not new passes.

What changes

• is_safe only adds passes; it never moves or removes one. Nothing that main blocks can pass, which is the property @lfengad suggested on fix(guardrail): fold code points before to_ascii erases them #193.
  • A new first pass checks the prompt as written, so the folding can work on it. The existing ASCII pass, the lemmatized pass and the exact-match pass all still run unchanged. A test pins the middle-dot case with a lemmatizer that turns dynamics into dynamic, so the second pass cannot rescue it.
  • For a non-ASCII prompt, the same passes also run on up to three other readings: invisible characters deleted, lookalike letters folded to Latin, and both together. An ASCII prompt never takes this path, so English prompts are unaffected by construction. A test pins that.
• The lookalike table (lookalikes.py) is generated from Unicode's confusables.txt 18.0.0 (UTS A2v sound encoder inference #39). It holds 728 code points whose skeleton is one Latin letter and that NFKC does not already fold. It is vendored, so there is no new dependency, and format_table regenerates it exactly.
  • A word is folded only when it looks like a disguised Latin word: it contains a Latin letter, or every letter in it is a lookalike.
  • Folding every listed letter in every word was measured in a prototype and rejected. It rewrites real Russian, Greek and Ukrainian text into Latin fragments that fuse into entries, and it blocked 9–11% of ordinary phrases in those languages. The per-word rule catches the same disguises (see the numbers below).
• Whitelisted phrases are exempted span by span. Matching runs on the whole prompt as before. A match is dropped only when it falls entirely inside an occurrence of a whitelisted phrase. Matches that straddle the phrase boundary stay blocked, so camera shot gun is still caught when camera shot is whitelisted.
  • Phrases are located before invisible characters are folded, so the zero-width form of the entry is not exempted.
  • Tokens keep " $ ' * @ when they are compared with a phrase, because better_profanity's leetspeak mapping uses those characters.
• The phrase mechanism is inert until a list owner adds a phrase. The ask is desk in.
• coverage.py writes five spellings of every entry and runs them through is_safe: verbatim, split by a space, split by a zero-width space, fullwidth, and lookalike. It is run before and after a list change, and --compare exits non-zero if anything that was blocked now passes. Results are keyed by index, so entry text is never printed.
• hygiene.py flags entries that split into two dictionary words, which is how ordinary prose collides with them. It reads lists with the production reader.
• One setup path. __init__, the tests, hygiene.py and coverage.py all load lists through Blocklist.load_lists / from_lists. Before this, there were four hand-copied versions, and the copies are where the bugs found on fix(guardrail): do not block prose whose words fuse into a blocklist entry #177 lived.

Measured on the shipped list

383 entries × 5 spellings = 1915 prompts, run through is_safe with the production lists.

             main 9214f43     this PR          this PR + whitelist "desk in"
             blocked/passed   blocked/passed   blocked/passed
verbatim     382 / 1          382 / 1          382 / 1
spaced       339 / 44         339 / 44         339 / 44
invisible    339 / 44         382 / 1          382 / 1
fullwidth     16 / 367        382 / 1          382 / 1
lookalike      2 / 381        382 / 1          382 / 1

coverage.py --compare main -> this PR:              lost 0, gained 789
coverage.py --compare this PR -> + "desk in":        lost 0, gained 0

Nothing that main blocks passes with this PR, checked prompt by prompt. Whitelisting desk in gives up none of the 1915 spellings.

The one entry that passes in every row also passes verbatim on main. The entry text itself contains punctuation, so the matcher can never produce it. That is a data issue in the list and is left to its owners. Which entry it is can be shared privately.

Cost on ordinary non-English text. Cosmos prompts are English, and an ASCII prompt is never touched. To bound the cost anyway, every run of three consecutive words from a 50,000-word frequency list per language (49,998 phrases each) was run through is_safe, on main and on this branch:

Language	Russian	Greek	Ukrainian	Persian	Arabic	Hebrew
Newly blocked	9 (0.02%)	66 (0.13%)	33 (0.07%)	0	0	0
Newly allowed	0	0	0	0	0	0

All of the new blocks come from the lookalike fold. The invisible-character reading adds none.

Known limits

• A normal space inside one word of a multi-word entry still passes (sn ow white), 44 entries, the same as main. Joining words back together would also join ordinary words, which is the false-positive problem from fix(guardrail): let the whitelist hold phrases, so ordinary prose stops matching blocklist entries #186. It needs its own measurement, so it is left for a follow-up.
• The exempted spelling itself passes. This is inherent to any fix that clears the false positive.
• Punctuation inside the phrase is not exempted. a desk, in the corner and a desk. In the background stay blocked even with desk in whitelisted, because the fused match spans the punctuation differently from the phrase. This errs strict.
• A word that has no Latin letter, and also has a letter with no Latin twin, is never folded, by the per-word rule above. That is what keeps ordinary Russian, Greek and Ukrainian text from being blocked.

Review points from #177, #186 and #193

Point	Where it was raised	Status
Add the raw pass instead of moving to_ascii; test boston·dynamics	#193	Done. The test fails without the change.
Tests must exercise is_safe, not censor_prompt	#193	Done. Normalization tests, phrase tests, lookalike and invisible tests, and the coverage sweep all use is_safe. Removing the raw pass fails the fullwidth and accent tests.
Masking removes tokens beyond the phrase (camera shot gun)	#186 L147	Replaced by span-based exemption. camera shot gun and in gun boundary tests added.
A probe built from the wrong list makes the gate unable to fail	#186 hygiene.py L260	Fixed. The probe checks what the gate advises.
# lines are skipped by the audit but loaded by production	#186 hygiene.py	Fixed. The audit uses the production reader.
The corpus check points the wrong way (new passes, not new blocks)	#186	coverage.py --compare
Middle dot on pass 1	#186 L303	Fixed by the additive pass
Test helpers re-implement __init__ bookkeeping	#177 L65	Fixed. One load_lists path for all callers.
string.punctuation overlaps the leetspeak characters	#177 L200	Applied to phrase matching
A phrase entry should be a data change signed off by list owners	#177, #186	The mechanism is inert. Adding desk in is left to the list owners.

Tests

• pytest cosmos_framework/auxiliary/guardrail/blocklist/: 99 passed. ruff check and ruff format --check (0.12.7) are clean.
• Mutation checks confirm the new tests fail without each fix:
  • Removing the alternate readings fails 7 lookalike tests.
  • Dropping the invisible-character reading fails 5 tests.
  • Folding every word instead of only the disguised ones fails the real-word test.
• The CPU CI jobs were run locally with the same commands and uv 0.11.29:
  • pre-commit, base config and main config: all hooks pass, and no files are modified.
  • Docs: 52 tests pass, and mkdocs build --strict is clean.
• The GPU smoke tests run with guardrails on, so is_safe runs on their prompts. All 14 example prompts under inputs/, defaults/ and examples/ pass on both main and this branch. They are all ASCII.

Ask for the list owners

The phrase mechanism does nothing until a phrase is added. The whitelist lives in the nvidia/Cosmos-Guardrail1 checkpoint, not in this repo. @lfengad, could you point us to the owners of that list? Two things are waiting on them:

• the phrase desk in, once this merges;
• the entry whose text contains punctuation.

Supersedes #193, #186 and #197.

🤖 Generated with Claude Code

[lfengad]

Review of 5119bef: the Unicode hardening fixes real blocklist bypasses and is worth landing. I would keep the feedback scoped as follows—two correctness fixes if the phrase/audit features remain in this PR, one follow-up gap, and one downstream dependency requirement.

1. Fix the phrase-exemption span calculation before merging phrase support.

censored_token_spans() assigns the total remaining token-count reduction to the current censored match. With multiple collapsed matches, it attributes later matches' swallowed tokens to the first match.

A static walkthrough gives this synthetic counterexample:

blocklist:  ["nicecamera", "shotgun"]
whitelist:  ["nice camera shot", "gun show"]
prompt:     "nice camera shot gun show"

The two matches occupy source spans (0,2) and (2,4); the helper infers (0,3) and (3,4). Both inferred spans fall inside whitelist phrases, so the cross-boundary shot gun match can be incorrectly exempted. Please obtain reliable match/source spans rather than reconstructing each width from the global token-count difference, and add an end-to-end is_safe() regression for multiple matches/phrases and boundary-crossing combinations. This matters when phrase whitelisting is configured; it is not a claim that the unchanged shipped whitelist already triggers this case.

2. Fix coverage comparison before using it as a no-regression gate.

measure() identifies prompts only by entry index and spelling. List edits or reordering can make the same key refer to a different prompt. The CLI ignores missing keys and returns success whenever lost is empty, including potentially incomparable result sets.

Please evaluate both versions against the same frozen probe corpus, retain stable opaque probe identities/corpus metadata, and fail or report “incomparable” if probe identities differ. Removing/moving an entry must not silently remove its old probes from the safety comparison. Alternatively, split this tool out rather than treating its current output as a reliable gate.

Follow-up, not a new regression: fullwidth/accent hardening still misses terms present only in exact_match. That check runs after to_ascii(), while alternate_spellings() omits the normalized-only reading. For example, with custom ["deskin"] and exact-match ["chromebook"], the fullwidth exact-only term loses its letters before the exact check. Include the normalized reading in the complete check chain, or explicitly scope and track this as a separate hardening follow-up.

I4/downstream backport: the production module now imports lookalikes.py. Carry that new helper and adapt its import together with the existing mapped files; do not assume an existing file manifest includes newly added production dependencies. This is a backport requirement, not a missing file in this public PR.

The phrase mechanism and audit tools can also be split out so the core Unicode fix can proceed independently. These findings are from static code/dependency analysis; I did not execute the project tests or a downstream run.