Skip to content

Resolve companion ranges across filename case differences - #19

Draft
imnasnainaec wants to merge 16 commits into
mainfrom
ranges-companion-filename-case
Draft

Resolve companion ranges across filename case differences#19
imnasnainaec wants to merge 16 commits into
mainfrom
ranges-companion-filename-case

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

A LIFT folder written on Windows can spell its pair inconsistently — Dict.LIFT beside Dict.lift-ranges, or the reverse — and load fine there, because the filesystem folds case. On a case-sensitive filesystem the sibling candidate is built from the .lift's own name, so it missed: the companion was skipped silently, and every range it defined went absent.

Zip packages inherit the fix: load_zip extracts and calls Lexicon.load, so _zip.py needs no change.

Matching a companion filename

A candidate that matches no file exactly falls back to one whose name folds onto it (_folded_matches).

  • Reached only after an exact miss, so it can only turn a miss into a hit — an exact match is always returned as-is. A case variant sitting beside an exactly named companion is not a collision, and the cost lands only where the lookup was already failing.
  • Case and normalization formcasefold() over NFC, not lower(). Decomposed filenames arrive from macOS, which stores and zips them that way, and neither NTFS nor ext4 folds the two forms.
  • Only the final path component, so an href under a directory spelled in another case still does not resolve. The hrefs this serves are bare basenames or same-folder relatives.
  • One directory read per folder, cached across a load's candidates.

Three things the fallback must not resolve to

Each is reachable once a miss can match something other than the exact name:

  • The .lift itself. A header href spelling Dict.lift beside a Dict.LIFT folds onto the lexicon, and RangesFile.load rejects a <lift> root — failing the whole load rather than skipping one companion.
  • A file beside a folder. An href of "" normalizes to the LIFT folder itself. Folding a folder's own name searches its parent, so any file up there spelled like the folder came back as the companion. A candidate that exists as a directory stops the lookup.
  • A companion already tracked. Path.resolve() leaves case alone on macOS, so one file reached under two spellings was loaded twice and written twice by save().

An ambiguous name resolves to nothing

Where several files in the folder fold onto one candidate, which of them the name meant is not recoverable — the filesystem that wrote them could not have told them apart. None is loaded, and validation reports the collision as a new ambiguous-ranges-file warning:

warning [ambiguous-ranges-file] Dict.LIFT: companion 'Dict.LIFT-ranges' matches
'Dict.Lift-ranges', 'Dict.lift-ranges'; they differ only in case or Unicode
normalization, so none of them is loaded
  • Every spelling named by code point, since two can differ only in normalization and render identically.
  • One finding per colliding group, however many candidate names fold onto it.
  • The sibling name is covered too — the case no header href reports on, and the one a folder authored on Windows most often collides over.

--strict promotes the warning to an error, so a CI run can refuse such a folder outright.

Validation and loading agree on what exists

dangling-ranges-href decides existence with the same lookup and the same refusals, so a companion spelled in another case is no longer reported missing on a case-sensitive filesystem, and an href resolving to the .lift still warns. Where a collision blocks the lookup both findings appear: one says why nothing resolved, the other which range went unmet. _same_file resolves its own arguments rather than trusting the caller's spelling, so the two sides cannot disagree about a path spelled two ways.

The candidate list moved out of _resolve_ranges into _ranges_candidates for validation to walk, and yields each path once — an href repeated across ranges, or agreeing with the sibling, was looked up as many times as it was written. Building the sibling candidate with with_name rather than with_suffix also keeps a .lift loaded under a name with no extension from raising, which parse_document has always accepted.

Tests

Twelve in test_ranges_folder.py. Two spellings of one name cannot coexist where the filesystem folds them together, so:

  • Three tests need a case-sensitive filesystem and skip elsewhere.
  • The collision test that runs everywhere separates its two companions by which of two accents is decomposed — the one difference NTFS and APFS keep — so the refusal, the warning, and the per-group deduplication are exercised on the Windows leg too. It skips only where normalization forms fold, as on macOS.

python scripts/check.py green: 581 passed, 3 skipped, 97.55% coverage.

Media hrefs have the same case problem and are deliberately untouched here — #34 carries the design question that makes them different from companions.

🤖 Generated with Claude Code


Devin review: https://app.devin.ai/review/sillsdev/python-sil-lift/pull/19


This change is Reviewable

imnasnainaec and others added 12 commits August 25, 2026 12:26
A LIFT folder written on Windows can spell its pair inconsistently —
Dict.LIFT beside Dict.lift-ranges, or the reverse — and load fine there,
because the filesystem folds case. On Linux the sibling candidate is built
from the .lift's own suffix, so it missed, the companion was skipped
without a word, and every range it defined went absent.

Candidates that match no file exactly now fall back to one whose name
differs only in case. The fallback is reached only after an exact miss, so
a case-folding filesystem never enters it and behaves as before; a
case-sensitive one gets one directory read per folder, cached across the
candidate list.

Where several names fold together the lexicographically first wins. The
choice is arbitrary but fixed, which matters more than which file it picks:
directory order varies between filesystems and runs, and a companion that
loads differently on consecutive reads would be worse than one that never
loads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.1.0 has not shipped, so there is no released behavior for an Unreleased
entry to be fixing — the tolerance is simply part of what companion
discovery does in the first release. Fold it into that bullet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The directory listing the fallback builds called its files "entries", the
word this module uses for a LIFT <entry> everywhere else — the same
collision that keeps byte regions from being called spans. Name them
files.

Spell the surrounding prose the way the rest of the package does: a
fallback that runs rather than fires, a name that matched no file rather
than missed, a helper named for the filesystem it probes rather than
abbreviating it, and fixture names deliberately not taken from the
corpus file. Unpack the two densest clauses so each reads in one pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Companion lookup folds names with casefold() over NFC rather than
lower(), so a Turkish-cased or NFD-spelled name resolves the way it does
on the filesystem that wrote it — FLEx mixes normalization forms within
one export. Ties break in code point order on every platform; sorting
Path objects left the choice to directory order on Windows, where
PurePath ordering is itself case-folded. An unstattable exact spelling
now falls through to the folded lookup instead of giving up.

A candidate that folds onto the .lift itself is skipped: RangesFile.load
rejects a <lift> root, so a header href naming the lexicon in another
case took the whole load down. One that folds onto a companion already
tracked is skipped too — Path.resolve() leaves case alone on macOS, so a
single file reached under two spellings was loaded and tracked twice,
and written twice by save().

The sibling candidate is built with with_name, which agrees with
with_suffix on every name that has an extension and does not raise on a
name without one. Nothing upstream requires the .lift extension:
parse_document never inspects it.

dangling-ranges-href decides existence with that same lookup, so a
companion spelled in another case is no longer reported missing on a
case-sensitive filesystem.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 0.1.0 entry said companion names fold on case alone; they fold on
Unicode normalization form as well. The folder guide listed the
candidates tried but never mentioned the folding at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three helpers each stated a rule, defended it, then disclaimed it.
What is left is the reasoning the code cannot show: casefold over lower,
NFC, why only the final component folds, why code point order, and what
the fold pre-check protects the inode comparison from.

The folder guide drops the folding sentence outright — it describes
behavior no reader acts on, in a paragraph otherwise about which
candidate wins. The 0.1.0 entry keeps the fact and loses the
justification, which now lives only in the docstrings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tion

A candidate that exists as a directory now stops the lookup instead of
falling through to the folded listing. An href of "" normalizes to the
LIFT folder itself and one of "sub/" to a subfolder, and folding a
folder's own name searches its *parent*: any file there spelled like the
folder was returned as the companion, and RangesFile.load then rejected
its root and failed the whole load.

dangling-ranges-href also treats a match that is the .lift itself as no
match. _resolve_ranges refuses to take the lexicon for its own
companion, so a header href folding onto it resolves to nothing that
supplies the range — the reference is as dangling as a missing file, and
went unreported on a case-sensitive filesystem.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_same_file compared the spellings it was handed, so a caller had to
canonicalize first to get a true answer. _resolve_ranges did; the
dangling-ranges-href check did not, and an href reaching the .lift
through a ".." segment or a symlink read there as some other file. The
loader skipped that candidate as self-referential while validation
counted it as a companion that exists, leaving the header range both
unsupplied and unreported.

Resolving inside _same_file makes the answer independent of how the
caller spelled its arguments, and retires the loader's own
pre-resolution — the duplicate that let the two drift apart. A path that
will not resolve now compares false instead of falling back to the
spelling as given; it would fail the samefile stat on the next line
regardless.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_fold cited FLEx's mixed NFC/NFD content as its reason to normalize
filenames. That is a different code path and no evidence for the names;
decomposed filenames arrive from macOS, so say that instead.

_existing_file described itself against the behavior it replaced, which
reads oddly once nothing remembers that behavior. Lexicon.load's
candidate list and its matching rules split into separate paragraphs,
and "every one that exists is loaded" becomes "every distinct file among
them" — candidates resolving to the lexicon, to a directory, or to a
file already tracked all exist and are deliberately skipped.

The dangling-ranges-href comment leads with what the check catches
rather than closing with it, and loses a restatement of _existing_file's
own docstring.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docstrings and comments only, plus one rename.

_existing_file claimed a case-folding filesystem never reaches the
fallback. That is false on NTFS, which folds case but not normalization:
an NFD companion misses the exact stat and only the fallback finds it.
The guarantee that does hold everywhere — an exact hit is returned
unchanged — leads instead. Its summary named the argument rather than
the return value, and its motivation read as though LIFT folders can
only be written on Windows, when what matters is whether the authoring
filesystem folds case, as macOS also does.

The with_name comment justified a choice against with_suffix rather than
warning about it. Naming the hazard is what stops someone reaching for
the tidier call and reintroducing the raise it avoids.

_case_sensitive_filesystem returned a bool under a noun phrase that
promises a filesystem; _same_file and _same_dir are the house pattern
for a predicate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Stable" names a specific property in sorting — preserving the relative
order of equal elements — which is close enough to what is meant here to
be read as a claim about the sort rather than about the outcome. The
choice among fold-equal names is deterministic: same folder, same winner,
on every platform and every run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Where the exact name missed and more than one file folded onto it, the
lookup took the first in code point order. That puts the conventionally
spelled Dict.lift-ranges last, behind any oddly cased twin beside it, and
picks between files whose contents can differ — a stale export, or a
rename git recorded as an add — with nothing said about it. Which one the
name meant is not recoverable, so none of them is loaded now.

Validation reports the collision as ambiguous-ranges-file, naming every
spelling by code point: two can differ only by normalization and render
identically. It walks the same candidates load does, so it covers the
sibling name as well, which no header href reports on and which is what a
folder authored on Windows most often collides over. One finding per
colliding group, however many candidates fold onto it; a dangling href
beside it says which range went unmet.

A name that matches a file exactly still resolves to it. Folding runs
only after that miss, so an odd-cased variant sitting beside an exactly
named companion is no collision at all.

The candidate list moves out of _resolve_ranges into _ranges_candidates
for validation to walk, and now yields each path once: an href repeated
across ranges, or agreeing with the sibling, was looked up as many times
as it was written.

A case-only collision cannot be created where the filesystem folds case,
and there the exact-name stat folds too — so the test that runs
everywhere separates its two companions by which of two accents is
decomposed, the one difference NTFS and APFS keep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@imnasnainaec
imnasnainaec force-pushed the ranges-companion-filename-case branch from 0f6fdce to 86c9013 Compare August 25, 2026 17:25
imnasnainaec and others added 4 commits August 25, 2026 13:29
NFC is a normalization form, not a dimension along which two filenames
differ: names collide because folding them applies NFC, and neither of
the two need be in that form — a stem with two accents has four spellings
that fold onto one key. So the problem code's description says the files
answer to one name under case folding and NFC, rather than that they
differ by it.

Two spellings differ *in* case or normalization, not *by* it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The problem-code reference told the reader how folding works and why the
collision is unresolvable. Both belong where the lookup lives; what a
reader needs here is that the ranges go absent and that leaving one file
brings them back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comments restated the lookup helpers' docstrings and described the
shape of the two loops below them. What is left is the part the code does
not show: that the check walks every candidate because a collision on the
sibling name has no href to report it, and why the spellings are named by
code point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A candidate folding onto several files warned even when another candidate
had named one of them exactly and loaded it — so the finding claimed that
none of them is loaded while one was, and pointed a reader at renaming
the very file supplying their ranges.

Reachable whenever the sibling name resolves and a header href is spelled
a third way: the sibling supplies the range, which also keeps
dangling-ranges-href quiet, so the false finding was the only thing said
about a folder that had lost nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🟩Low Low-priority PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant