Skip to content

Drop per-entry allocations in ListDir - #42

Merged
andrew merged 3 commits into
mainfrom
perf/listdir-allocations
Sep 13, 2026
Merged

andrew merged 3 commits into
mainfrom
perf/listdir-allocations

Conversation

@andrew

@andrew andrew commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

isInDir re-normalized dirPath on every call even though both callers (tar.go and zip.go ListDir) normalize once before the loop, used strings.Split to check for a slash, and concatenated filePath+"/" to run HasPrefix. ListDir used strings.Split to read the first path segment and built the full subdir path before the seenDirs dedup lookup. All of that ran once per archive entry per ListDir call.

isInDir now requires a normalized dirPath and uses IndexByte/HasPrefix with no allocation. ListDir keys seenDirs by the child segment name (a substring of the entry name), so the full subdir path is only built once per distinct child.

Adds BenchmarkListDir (2000-entry tar, 20 subdirs). Interleaved benchstat, arm64 darwin, n=10:

                    │   before    │             after              │
                    │   sec/op    │   sec/op     vs base           │
ListDir/#00           195.67µ ± 9%   33.11µ ± 4%  -83.08% (p=0.000)
ListDir/lib           174.19µ ± 9%   39.06µ ± 2%  -77.57% (p=0.000)
ListDir/lib/sub00      74.00µ ± 6%   16.80µ ± 3%  -77.29% (p=0.000)
                    │  allocs/op  │  allocs/op   vs base           │
ListDir/#00            6003.0 ± 0%     4.0 ± 0%   -99.93% (p=0.000)
ListDir/lib            6014.0 ± 0%    34.0 ± 0%   -99.43% (p=0.000)
ListDir/lib/sub00      2011.0 ± 0%    11.0 ± 0%   -99.45% (p=0.000)

BenchmarkTarBrowse cases unchanged in sec/op (all p>0.05), −4 allocs/op each.

Also fixes a pre-existing duplicate: an archive that lists files before their parent directory entry got a synthetic subdir entry from the first file and then the explicit entry appended as well. TestZipListDirNoDuplicatesWithExplicitDirEntries only covered the dir-entry-first ordering (GitHub zipball shape); adds TestListDirNoDuplicatesWithLateDirEntry for the reversed order.

isInDir now requires a normalized dirPath (both callers already pass
one) and uses IndexByte instead of Split; ListDir's subdir check does
the same. No behaviour change.
The full path was built for every entry to use as the dedup key; the
child segment name is just as unique within a single ListDir call and
is a substring of the entry name, so the concat now happens once per
distinct child rather than once per entry.
An archive that lists files before their parent directory entry got a
synthetic subdir entry from the first file and then the explicit entry
appended as well. Check seenDirs before appending an explicit dir entry
so the first one wins, matching the dir-entry-first case.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Resolve directory-key collisions and preserve explicit directory metadata in both TAR and ZIP implementations.

Pull request overview

Optimizes archive directory listing by removing per-entry allocations and improving directory deduplication.

Changes:

  • Reworks normalized path checks and child-directory tracking.
  • Adds ListDir benchmark coverage.
  • Adds late-directory-entry regression coverage.
File summaries
File Reviewed changes
zip.go Optimizes ZIP directory listing and deduplication.
tar.go Optimizes TAR directory listing and deduplication.
tar_bench_test.go Adds ListDir benchmarks.
archives.go Updates normalized path membership checks.
archives_test.go Expands path and duplicate-entry tests.
Review details

Suppressed comments (5)

tar.go:175

  • Because isInDir also accepts the requested directory entry itself, a valid archive can reach this block with path == strings.TrimSuffix(dirPath, "/") (for example, a TAR directory named dir without a trailing slash). TrimPrefix(path, dirPath) then leaves dir, so this key collides with a real child named dir under the requested directory and suppresses that child later. Treat the exact directory entry as an empty relative name (or use a collision-proof key) before deduplicating.
				name := strings.TrimSuffix(strings.TrimPrefix(path, dirPath), "/")
				if seenDirs[name] {
					continue

tar.go:176

  • When the first entry under a directory is a file, the synthetic directory is recorded before a later explicit directory entry arrives. This continue then discards the explicit entry's Mode, ModTime, and HasMode, so ListDir returns different metadata depending on archive entry order. Track the synthetic entry's index and replace it with the real directory FileInfo when the explicit entry is encountered.
				if seenDirs[name] {
					continue
				}

zip.go:176

  • Because isInDir also accepts the requested directory entry itself, a valid archive can reach this block with path == strings.TrimSuffix(dirPath, "/") (for example, a ZIP directory marked as a directory without a trailing slash). TrimPrefix(path, dirPath) then leaves dir, so this key collides with a real child named dir under the requested directory and suppresses that child later. Treat the exact directory entry as an empty relative name (or use a collision-proof key) before deduplicating.
				name := strings.TrimSuffix(strings.TrimPrefix(path, dirPath), "/")
				if seenDirs[name] {
					continue

zip.go:177

  • When the first entry under a directory is a file, the synthetic directory is recorded before a later explicit directory entry arrives. This continue then discards the explicit entry's Mode, ModTime, and HasMode, so ListDir returns different metadata depending on archive entry order. Track the synthetic entry's index and replace it with the real directory FileInfo when the explicit entry is encountered.
				if seenDirs[name] {
					continue
				}

zip.go:178

  • The late-directory ordering case is only exercised through tarReader; the same deduplication branch was changed here for ZIP, but the existing ZIP test covers only the directory-entry-first order. Add an equivalent ZIP fixture/test with a child file before its explicit parent so this regression cannot return in the ZIP implementation.
			if f.FileInfo().IsDir() {
				name := strings.TrimSuffix(strings.TrimPrefix(path, dirPath), "/")
				if seenDirs[name] {
					continue
				}
				seenDirs[name] = true
  • Files reviewed: 5/5 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@andrew
andrew merged commit 5dd7996 into main Sep 13, 2026
6 checks passed
@andrew
andrew deleted the perf/listdir-allocations branch September 13, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants