Drop per-entry allocations in ListDir - #42
Merged
Merged
Conversation
isInDir now requires a normalized dirPath (both callers already pass one) and uses IndexByte instead of Split; ListDir's subdir check does the same. No behaviour change.
The full path was built for every entry to use as the dedup key; the child segment name is just as unique within a single ListDir call and is a substring of the entry name, so the concat now happens once per distinct child rather than once per entry.
An archive that lists files before their parent directory entry got a synthetic subdir entry from the first file and then the explicit entry appended as well. Check seenDirs before appending an explicit dir entry so the first one wins, matching the dir-entry-first case.
There was a problem hiding this comment.
🔵 Needs a closer look
Resolve directory-key collisions and preserve explicit directory metadata in both TAR and ZIP implementations.
Pull request overview
Optimizes archive directory listing by removing per-entry allocations and improving directory deduplication.
Changes:
- Reworks normalized path checks and child-directory tracking.
- Adds
ListDirbenchmark coverage. - Adds late-directory-entry regression coverage.
File summaries
| File | Reviewed changes |
|---|---|
zip.go |
Optimizes ZIP directory listing and deduplication. |
tar.go |
Optimizes TAR directory listing and deduplication. |
tar_bench_test.go |
Adds ListDir benchmarks. |
archives.go |
Updates normalized path membership checks. |
archives_test.go |
Expands path and duplicate-entry tests. |
Review details
Suppressed comments (5)
tar.go:175
- Because
isInDiralso accepts the requested directory entry itself, a valid archive can reach this block withpath == strings.TrimSuffix(dirPath, "/")(for example, a TAR directory nameddirwithout a trailing slash).TrimPrefix(path, dirPath)then leavesdir, so this key collides with a real child nameddirunder the requested directory and suppresses that child later. Treat the exact directory entry as an empty relative name (or use a collision-proof key) before deduplicating.
name := strings.TrimSuffix(strings.TrimPrefix(path, dirPath), "/")
if seenDirs[name] {
continue
tar.go:176
- When the first entry under a directory is a file, the synthetic directory is recorded before a later explicit directory entry arrives. This
continuethen discards the explicit entry'sMode,ModTime, andHasMode, soListDirreturns different metadata depending on archive entry order. Track the synthetic entry's index and replace it with the real directoryFileInfowhen the explicit entry is encountered.
if seenDirs[name] {
continue
}
zip.go:176
- Because
isInDiralso accepts the requested directory entry itself, a valid archive can reach this block withpath == strings.TrimSuffix(dirPath, "/")(for example, a ZIP directory marked as a directory without a trailing slash).TrimPrefix(path, dirPath)then leavesdir, so this key collides with a real child nameddirunder the requested directory and suppresses that child later. Treat the exact directory entry as an empty relative name (or use a collision-proof key) before deduplicating.
name := strings.TrimSuffix(strings.TrimPrefix(path, dirPath), "/")
if seenDirs[name] {
continue
zip.go:177
- When the first entry under a directory is a file, the synthetic directory is recorded before a later explicit directory entry arrives. This
continuethen discards the explicit entry'sMode,ModTime, andHasMode, soListDirreturns different metadata depending on archive entry order. Track the synthetic entry's index and replace it with the real directoryFileInfowhen the explicit entry is encountered.
if seenDirs[name] {
continue
}
zip.go:178
- The late-directory ordering case is only exercised through
tarReader; the same deduplication branch was changed here for ZIP, but the existing ZIP test covers only the directory-entry-first order. Add an equivalent ZIP fixture/test with a child file before its explicit parent so this regression cannot return in the ZIP implementation.
if f.FileInfo().IsDir() {
name := strings.TrimSuffix(strings.TrimPrefix(path, dirPath), "/")
if seenDirs[name] {
continue
}
seenDirs[name] = true
- Files reviewed: 5/5 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
isInDirre-normalizeddirPathon every call even though both callers (tar.goandzip.goListDir) normalize once before the loop, usedstrings.Splitto check for a slash, and concatenatedfilePath+"/"to runHasPrefix.ListDirusedstrings.Splitto read the first path segment and built the full subdir path before theseenDirsdedup lookup. All of that ran once per archive entry perListDircall.isInDirnow requires a normalizeddirPathand usesIndexByte/HasPrefixwith no allocation.ListDirkeysseenDirsby the child segment name (a substring of the entry name), so the full subdir path is only built once per distinct child.Adds
BenchmarkListDir(2000-entry tar, 20 subdirs). Interleaved benchstat, arm64 darwin, n=10:BenchmarkTarBrowsecases unchanged in sec/op (all p>0.05), −4 allocs/op each.Also fixes a pre-existing duplicate: an archive that lists files before their parent directory entry got a synthetic subdir entry from the first file and then the explicit entry appended as well.
TestZipListDirNoDuplicatesWithExplicitDirEntriesonly covered the dir-entry-first ordering (GitHub zipball shape); addsTestListDirNoDuplicatesWithLateDirEntryfor the reversed order.