Skip to content

One document-name derivation, shared by every output path #42

Description

@charitarthchugh

The problem

paperscale derives an output name from a Record's Source-File in more than one place, and after #41 those places no longer agree.

path rule collision behaviour
get_markdown_path() (src/paperscale/pipeline.py:674) replaces the extension silently overwrites (#32); a guard is being added, so it will stop loudly instead
embed's .npz Sink (#22, #41) appends the extension cannot collide on extension; anything else is fatal at startup
embed's LanceDB document_name (#27) same as .npz same

#32 chose the guard over the rename deliberately — renaming every existing markdown file is a larger change than that bug warrants, and a guard matches the DuplicateSourceFileError precedent this repo already follows. That is the right near-term call. It leaves two things unfixed.

1. A mixed corpus still cannot be exported. The guard turns a silent overwrite into a hard stop. --pdfs accepts "Local PDF/image paths", so case.pdf and case.tiff in one directory is a supported input that now fails instead of losing data. Better, not fixed.

2. The two derivations have diverged. #22 chose extension-replacement specifically to mirror get_markdown_path rather than diverge from it. #41 broke that mirror on purpose, because the thing being mirrored is the defect. So the stated reason for a shared rule still stands while nothing actually shares one.

Three collision classes; only one is addressed anywhere

class example prevented by extension-append?
extension replacement case.pdf + case.tiff yes
sanitization — lstrip("/"), .. and empty components dropped /a/case.pdf + a/case.pdf no
tarball collapse x.tar.gz::doc.pdf + x.tar::doc.pdf no

Both remaining classes are fatal-at-startup in embed and silent in the markdown export.

What this ticket would do

  • One derivation function, used by the markdown export, the .npz Sink and LanceDB.
  • A migration decision. Changing the markdown rule renames every existing export. Markdown is regenerable from results/*.jsonl via --export-markdown, so the cost is one command plus clearing the old tree — but it is still a breaking change to shipped output paths and should be announced rather than slipped in.
  • One collision policy across all three outputs, covering all three classes.

Not urgent — measured

The live corpus is 39,905 files, all .pdf, with zero collisions of any class. The 49-Document smoke set derives 49 distinct names. Nothing is losing data today. This is debt to pay deliberately, not a fire.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions