Skip to content

Resolve less per extracted file in the CLI - #98

Merged
bitplane merged 2 commits into
bitplane:masterfrom
tiendc:cli-extract-speed
Oct 25, 2026
Merged

bitplane merged 2 commits into
bitplane:masterfrom
tiendc:cli-extract-speed

Conversation

@tiendc

@tiendc tiendc commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

rars extract is slow per file rather than per byte. Under strace on Linux it makes about 92 system calls per extracted file, where UnRAR makes about 18. Most of those come from the verified-extraction checks. This PR keeps every check, with the same inputs and at the same points, and drops three places that redid work whose result could not have changed.

  • The destination was computed twice in a row. open_entry_named, and its RAR5 and redirection variants, computed the checked output path twice back to back. Only an in-memory reservation ran in between, so lstat walked every component of the path twice for nothing.
  • The staging root was resolved on every call. reject_alias canonicalized the private staging root each time it ran, twice per member. It is now resolved once, on the first alias check. Temporary storage used only for cross-device publication does not resolve its root.
  • Missing destinations were resolved in full before failing. For a destination that does not exist yet, which is most of them, canonicalize resolved every component before failing with NotFound. That happened in reject_alias, and again in the duplicate-identity check before publishing.
    • On Unix, one lstat now answers "missing" first. Other errors still go to canonicalize, which reports them as before.
    • reject_alias returns the destination's resolved path when it exists, and the identity check uses it instead of resolving the path again.
    • On Windows, a failed resolve is already a single failed open, so the lstat would only add a call; the shortcut is Unix-only.

On Linux this takes extraction from 92 to 68 system calls per file for RAR5, and from 97 to 72 for RAR 2.9 LZ (strace, 221 files). readlinkat calls fall by 60%.

Hosted runners, rars --threads 1 extract against master, three rounds in turn on the same archives (2,300–2,700 files, 130–160 MB, written by rars), MB/s:

master this PR
Windows, RAR 2.9 LZ 53–64 63–71
Windows, RAR5 45–46 49–51
Linux, RAR 2.9 LZ 190–193 222–224
Linux, RAR5 96–97 102–103
macOS, RAR 2.9 LZ 98–109 126–134
macOS, RAR5 77–81 87–88

Decoding alone (rars test) is unchanged. On an M1 Pro, extracting 8,869 files goes from 23–24 to 26–27 MB/s (RAR5), and from 25–26 to 28–30 (RAR 2.9 LZ).

Initial author validation:

  • cargo test -p rars-cli: 333 tests pass, including the two real-terminal tests (with an arm64 python3).
  • clippy -D warnings and fmt are clean.
  • The full CI workflow passes on my fork.

What is left is larger, and I'd rather ask before touching it. On Windows, 7-Zip still extracts those archives about three times faster (172–182 MB/s for RAR 2.9 LZ, 134–136 for RAR5). I timed the remaining per-file steps on their own; in ms per 1,000 files on a hosted Windows runner:

step ms per 1,000 files
create, write and close only 181
lstat on every component, four times +249
canonicalize, five times +290
staging, then hard link and unlink +122

Creating files relative to directory handles that are opened once, without following links (openat-style, for example through cap-std), costs the same as the plain create on Linux, macOS and Windows (185 against 181 ms on Windows). It also closes the window between checking a path and opening it.

If that direction is of interest, I can sketch it for the verified-extraction path in an issue first.

🤖 Generated with Claude Code

Maintainer review follow-up:

  • Cache the staging identity lazily so cross-device publication keeps its existing behavior without an extra path-resolution dependency.
  • Add four direct regression tests for existing/missing destination identities, Unix staging aliases with missing descendants, dangling links and non-directory ancestors, loop/permission errors, and retry after an initial resolution failure.
  • Local CLI suite: 359 tests passed, including terminal tests and external-decoder compatibility. Formatting is clean.
  • On the same 128-file nested fixtures, final RAR 2.9/RAR5 extraction used about 32% fewer syscalls for fresh output and 23% fewer for overwrites than master. Readlink/statx counts match the original PR; every extracted file was verified byte-for-byte. These measurements cover ordinary extraction, not cross-device publication.
  • Independent Claude reviews approved the original patch, follow-up plan, and completed changes with no blockers.

tiendc and others added 2 commits October 25, 2026 08:47
The CLI's verified extraction checks every member's destination before it
is staged and again before it is published. Three parts of that redid work
whose result could not have changed:

- open_entry_named (and its RAR5 and redirection variants) computed the
  checked output path twice in a row, with only an in-memory reservation
  between, walking every component with lstat both times.
- reject_alias resolved the private staging root on every call. It is now
  resolved once, when the storage is created.
- For a destination that does not exist yet, which is most of them,
  canonicalize walked every component before failing, in reject_alias and
  again in the duplicate-identity check before publishing. On Unix an lstat
  now answers "missing" first, and reject_alias hands the destination's
  resolved path, when it exists, to the identity check. On Windows a failed
  resolve is already one failed open, so nothing changes there.

Every check still runs with the same inputs, at the same points.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…them

Cache the resolved staging root on the first alias check instead of during
directory creation. Cross-device publication-only staging keeps its original
behavior without an unnecessary canonicalize call or new resolution failure.

Add direct coverage for existing destination identities, missing descendants
through staging aliases, dangling links, non-directory ancestors, loop and
permission errors, and retry after failed initial root resolution.

The ordinary extraction path retains the PR's syscall savings. Both the plan
and completed change received an independent Claude review with no blockers.
@bitplane
bitplane merged commit b2d6880 into bitplane:master Oct 25, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants