Five places in src/ implement something already written elsewhere: three
duplicate a standard-library capability, two duplicate each other. Each is
independent of the others; none is urgent. Blob links are pinned to d7e2166.
The stdlib already covers it
_scan.py walks bytes for offsets expat reports directly
The module
docstring
says "lxml exposes no byte offsets, so this module walks the raw bytes with a
small state machine". That is true of lxml and false of the stdlib:
xml.parsers.expat reports CurrentByteIndex for the current event's markup.
Against a document with a leading comment, a self-closing entry, and a CDATA
section containing a literal </entry>, ~20 lines of handler code produce the
whole of ScanResult:
header start= 75 end= 92 b'<header><ranges/>'
entry start= 102 end= 188 b'<entry id="a">...</lexical-unit>'
entry start= 197 end= 212 b'<entry id="b"/>'
entry start= 230 end= 264 b'<entry id="c"><![CDATA[</entry>]]>'
_skip_comment, _skip_cdata, _skip_pi, _skip_element, and the prolog
walk — roughly 100 of the module's 189 lines — have nothing left to do.
_skip_tag stays: the end-element event reports the offset of </, but a
self-closing element reports the same index for both of its events (197 above),
so finding its > still needs quote-aware scanning. To keep the current
refusals, set SetParamEntityParsing(XML_PARAM_ENTITY_PARSING_NEVER) and bail
from StartDoctypeDeclHandler.
This module underpins the byte-identity guarantee, so the corpus round-trip
tests are what make the swap safe to attempt at all. Worth weighing before
starting.
_nearest_entry scans a sorted list
_nearest_entry
walks entry_lines for the greatest line at or below a schema error's line, and
is called once per schema
error.
entry_lines is built in document order, so it is already sorted and
bisect_right applies. Today a large export that fails schema validation on
every entry costs entries x errors.
duplicate-form-lang counts by rescanning
The
check
is {lang for lang in langs if langs.count(lang) > 1} — a quadratic scan for
what collections.Counter does in one pass. Form counts per multitext are
small, so this is about saying it plainly rather than about speed; _writer.py
already imports Counter.
It duplicates itself
The subsense walk, five times
The same stack.pop() depth-first walk over sense.subsenses appears in
_cli._iter_senses,
_cli._iter_leaf_senses
(leaves only),
Entry.gloss_langs,
Lexicon.media_refs,
and
_validate._iter_senses.
One traversal on the model covers all five, and walking a sense tree is
something a caller wants anyway — it is a candidate for the public surface
rather than a private helper.
Three identical dataclass walkers
_iter_multitexts,
_iter_traits,
and
_iter_grammatical_infos
are the same recursion over dataclasses.fields with the isinstance target
swapped. One _iter_instances(obj, cls) replaces them; only _iter_multitexts
needs anything extra, since it also yields the field label.
Five places in
src/implement something already written elsewhere: threeduplicate a standard-library capability, two duplicate each other. Each is
independent of the others; none is urgent. Blob links are pinned to d7e2166.
The stdlib already covers it
_scan.pywalks bytes for offsets expat reports directlyThe module
docstring
says "lxml exposes no byte offsets, so this module walks the raw bytes with a
small state machine". That is true of lxml and false of the stdlib:
xml.parsers.expatreportsCurrentByteIndexfor the current event's markup.Against a document with a leading comment, a self-closing entry, and a CDATA
section containing a literal
</entry>, ~20 lines of handler code produce thewhole of
ScanResult:_skip_comment,_skip_cdata,_skip_pi,_skip_element, and the prologwalk — roughly 100 of the module's 189 lines — have nothing left to do.
_skip_tagstays: the end-element event reports the offset of</, but aself-closing element reports the same index for both of its events (197 above),
so finding its
>still needs quote-aware scanning. To keep the currentrefusals, set
SetParamEntityParsing(XML_PARAM_ENTITY_PARSING_NEVER)and bailfrom
StartDoctypeDeclHandler.This module underpins the byte-identity guarantee, so the corpus round-trip
tests are what make the swap safe to attempt at all. Worth weighing before
starting.
_nearest_entryscans a sorted list_nearest_entrywalks
entry_linesfor the greatest line at or below a schema error's line, andis called once per schema
error.
entry_linesis built in document order, so it is already sorted andbisect_rightapplies. Today a large export that fails schema validation onevery entry costs entries x errors.
duplicate-form-langcounts by rescanningThe
check
is
{lang for lang in langs if langs.count(lang) > 1}— a quadratic scan forwhat
collections.Counterdoes in one pass. Form counts per multitext aresmall, so this is about saying it plainly rather than about speed;
_writer.pyalready imports
Counter.It duplicates itself
The subsense walk, five times
The same
stack.pop()depth-first walk oversense.subsensesappears in_cli._iter_senses,_cli._iter_leaf_senses(leaves only),
Entry.gloss_langs,Lexicon.media_refs,and
_validate._iter_senses.One traversal on the model covers all five, and walking a sense tree is
something a caller wants anyway — it is a candidate for the public surface
rather than a private helper.
Three identical dataclass walkers
_iter_multitexts,_iter_traits,and
_iter_grammatical_infosare the same recursion over
dataclasses.fieldswith theisinstancetargetswapped. One
_iter_instances(obj, cls)replaces them; only_iter_multitextsneeds anything extra, since it also yields the field label.