Skip to content

Four hand-rolled implementations the stdlib covers, and three that duplicate each other #39

Description

@imnasnainaec

Five places in src/ implement something already written elsewhere: three
duplicate a standard-library capability, two duplicate each other. Each is
independent of the others; none is urgent. Blob links are pinned to d7e2166.

The stdlib already covers it

_scan.py walks bytes for offsets expat reports directly

The module
docstring

says "lxml exposes no byte offsets, so this module walks the raw bytes with a
small state machine". That is true of lxml and false of the stdlib:
xml.parsers.expat reports CurrentByteIndex for the current event's markup.
Against a document with a leading comment, a self-closing entry, and a CDATA
section containing a literal </entry>, ~20 lines of handler code produce the
whole of ScanResult:

header   start=  75 end=  92  b'<header><ranges/>'
entry    start= 102 end= 188  b'<entry id="a">...</lexical-unit>'
entry    start= 197 end= 212  b'<entry id="b"/>'
entry    start= 230 end= 264  b'<entry id="c"><![CDATA[</entry>]]>'

_skip_comment, _skip_cdata, _skip_pi, _skip_element, and the prolog
walk — roughly 100 of the module's 189 lines — have nothing left to do.
_skip_tag stays: the end-element event reports the offset of </, but a
self-closing element reports the same index for both of its events (197 above),
so finding its > still needs quote-aware scanning. To keep the current
refusals, set SetParamEntityParsing(XML_PARAM_ENTITY_PARSING_NEVER) and bail
from StartDoctypeDeclHandler.

This module underpins the byte-identity guarantee, so the corpus round-trip
tests are what make the swap safe to attempt at all. Worth weighing before
starting.

_nearest_entry scans a sorted list

_nearest_entry
walks entry_lines for the greatest line at or below a schema error's line, and
is called once per schema
error
.
entry_lines is built in document order, so it is already sorted and
bisect_right applies. Today a large export that fails schema validation on
every entry costs entries x errors.

duplicate-form-lang counts by rescanning

The
check

is {lang for lang in langs if langs.count(lang) > 1} — a quadratic scan for
what collections.Counter does in one pass. Form counts per multitext are
small, so this is about saying it plainly rather than about speed; _writer.py
already imports Counter.

It duplicates itself

The subsense walk, five times

The same stack.pop() depth-first walk over sense.subsenses appears in
_cli._iter_senses,
_cli._iter_leaf_senses
(leaves only),
Entry.gloss_langs,
Lexicon.media_refs,
and
_validate._iter_senses.
One traversal on the model covers all five, and walking a sense tree is
something a caller wants anyway — it is a candidate for the public surface
rather than a private helper.

Three identical dataclass walkers

_iter_multitexts,
_iter_traits,
and
_iter_grammatical_infos
are the same recursion over dataclasses.fields with the isinstance target
swapped. One _iter_instances(obj, cls) replaces them; only _iter_multitexts
needs anything extra, since it also yields the field label.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions