The problem
paperscale derives an output name from a Record's Source-File in more than one place, and after #41 those places no longer agree.
| path |
rule |
collision behaviour |
get_markdown_path() (src/paperscale/pipeline.py:674) |
replaces the extension |
silently overwrites (#32); a guard is being added, so it will stop loudly instead |
embed's .npz Sink (#22, #41) |
appends the extension |
cannot collide on extension; anything else is fatal at startup |
embed's LanceDB document_name (#27) |
same as .npz |
same |
#32 chose the guard over the rename deliberately — renaming every existing markdown file is a larger change than that bug warrants, and a guard matches the DuplicateSourceFileError precedent this repo already follows. That is the right near-term call. It leaves two things unfixed.
1. A mixed corpus still cannot be exported. The guard turns a silent overwrite into a hard stop. --pdfs accepts "Local PDF/image paths", so case.pdf and case.tiff in one directory is a supported input that now fails instead of losing data. Better, not fixed.
2. The two derivations have diverged. #22 chose extension-replacement specifically to mirror get_markdown_path rather than diverge from it. #41 broke that mirror on purpose, because the thing being mirrored is the defect. So the stated reason for a shared rule still stands while nothing actually shares one.
Three collision classes; only one is addressed anywhere
| class |
example |
prevented by extension-append? |
| extension replacement |
case.pdf + case.tiff |
yes |
sanitization — lstrip("/"), .. and empty components dropped |
/a/case.pdf + a/case.pdf |
no |
| tarball collapse |
x.tar.gz::doc.pdf + x.tar::doc.pdf |
no |
Both remaining classes are fatal-at-startup in embed and silent in the markdown export.
What this ticket would do
- One derivation function, used by the markdown export, the
.npz Sink and LanceDB.
- A migration decision. Changing the markdown rule renames every existing export. Markdown is regenerable from
results/*.jsonl via --export-markdown, so the cost is one command plus clearing the old tree — but it is still a breaking change to shipped output paths and should be announced rather than slipped in.
- One collision policy across all three outputs, covering all three classes.
Not urgent — measured
The live corpus is 39,905 files, all .pdf, with zero collisions of any class. The 49-Document smoke set derives 49 distinct names. Nothing is losing data today. This is debt to pay deliberately, not a fire.
Related
The problem
paperscale derives an output name from a Record's
Source-Filein more than one place, and after #41 those places no longer agree.get_markdown_path()(src/paperscale/pipeline.py:674)embed's.npzSink (#22, #41)embed's LanceDBdocument_name(#27).npz#32 chose the guard over the rename deliberately — renaming every existing markdown file is a larger change than that bug warrants, and a guard matches the
DuplicateSourceFileErrorprecedent this repo already follows. That is the right near-term call. It leaves two things unfixed.1. A mixed corpus still cannot be exported. The guard turns a silent overwrite into a hard stop.
--pdfsaccepts "Local PDF/image paths", socase.pdfandcase.tiffin one directory is a supported input that now fails instead of losing data. Better, not fixed.2. The two derivations have diverged. #22 chose extension-replacement specifically to mirror
get_markdown_pathrather than diverge from it. #41 broke that mirror on purpose, because the thing being mirrored is the defect. So the stated reason for a shared rule still stands while nothing actually shares one.Three collision classes; only one is addressed anywhere
case.pdf+case.tifflstrip("/"),..and empty components dropped/a/case.pdf+a/case.pdfx.tar.gz::doc.pdf+x.tar::doc.pdfBoth remaining classes are fatal-at-startup in
embedand silent in the markdown export.What this ticket would do
.npzSink and LanceDB.results/*.jsonlvia--export-markdown, so the cost is one command plus clearing the old tree — but it is still a breaking change to shipped output paths and should be announced rather than slipped in.Not urgent — measured
The live corpus is 39,905 files, all
.pdf, with zero collisions of any class. The 49-Document smoke set derives 49 distinct names. Nothing is losing data today. This is debt to pay deliberately, not a fire.Related
embedname rule and its "mirror the export" rationaleembed