feat(alias): add optional cache and index-assisted resolution#2608
Open
syscc wants to merge 3 commits into
Open
feat(alias): add optional cache and index-assisted resolution#2608syscc wants to merge 3 commits into
syscc wants to merge 3 commits into
Conversation
Co-authored-by: OpenAI Codex <codex@openai.com>
Member
|
Just a note: GitHub has Draft PRs for cases where the work isn’t ready for review yet, so you don’t need to wait until everything is review-ready before opening a PR. |
Contributor
Author
|
Thanks for the note. I’ll use Draft PRs for work-in-progress changes next time. |
fdb3ae7 to
de0b9ed
Compare
Contributor
Author
|
已补充本次 Alias 缓存与父目录预热更新,测试通过。麻烦管理审核一下,没问题的话帮忙合并。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / 摘要
Adds opt-in caching for the Alias driver to reduce repeated backend polling when listing aggregated directories and resolving virtual Alias paths to backend paths.
Latest updates in this PR:
Adds
alias_cache_enabled,alias_cache_expiration,alias_cache_max_entries, andalias_list_concurrencyAlias options.Caches aggregated Alias directory listings and resolved backend paths when enabled.
Treats missing Alias backend roots / future placeholder mounts as cacheable misses, so unavailable configured paths do not prevent caching valid results from existing backends.
Uses backend directory paths as the list cache key so internal
op.List(..., model.ListArgs{})calls withoutReqPathdo not collide.Avoids caching partial list results when any backend returns a non-not-found error.
Clears the Alias cache on write-like operations and invalidates refreshed list/resolved entries when
refreshis requested.Adds index-assisted Alias path resolution: when the search index is enabled, complete, and not currently building, Alias uses indexed parent/name entries to prioritize likely backend paths, then verifies the chosen backend path with
fs.Getbefore caching.Narrows cached resolved child backend paths to paths that actually exist in listed backend directories, reducing later scans for large Alias mounts.
Adds background parent-directory prewarming after direct file resolution. Directly opening
/videos/.../Season 1/E01can prewarm/videos/.../Season 1, so later sibling episode lookups benefit from the same Alias list/resolved cache.Keeps the feature disabled by default.
alias_cache_max_entries=0means unlimited entries.This PR has breaking changes.
/ 此 PR 包含破坏性变更。
This PR changes public API, config, storage format, or migration behavior.
/ 此 PR 修改了公开 API、配置、存储格式或迁移行为。
This PR requires corresponding changes in related repositories.
/ 此 PR 需要关联仓库同步修改。
Related repository PRs / 关联仓库 PR:
Related Issues / 关联 Issue
N/A
Testing / 测试
go test ./...go test ./drivers/aliasgo run . --data data --config /tmp/openlist-config-with-s3.json --log-std servergofmt -w drivers/alias/driver.go drivers/alias/cache.go drivers/alias/index.go drivers/alias/meta.go drivers/alias/util.go internal/search/search.gogo list ./drivers/alias ./internal/search ./internal/search/db ./internal/search/db_non_full_text ./internal/search/meilisearchgit diff --checkMissing backend roots / placeholder mounts
NAS Alias mount
/videos,alias_list_concurrency=8, cache enabled. The Alias config contains many future placeholder roots that currently do not exist, for example/ODS/mov20/电影and/ODS/tv65/电视剧./videos/电影/外语电影refresh=true/videos/电影/外语电影/videos/电影/外语电影/videos/电影refresh=true/videos/电影/videos/电影/videos/电视剧/欧美剧refresh=true/videos/电视剧/欧美剧/videos/电视剧/欧美剧Result: missing configured backend roots no longer prevent caching valid aggregated Alias list results.
Third-level directory cache behavior
After listing parent directories, cached child resolution only keeps backend paths that actually exist for that child name.
/videos/电视剧/videos/电视剧/videos/电视剧/国产剧/videos/电视剧/国产剧/videos/电视剧/日韩剧/videos/电视剧/日韩剧/videos/电影/华语电影/videos/电影/华语电影Direct file playback parent prewarm
Validated on beta image from this PR branch:
Directly resolving the first episode prewarms the parent season directory in the background. After the prewarm delay, the parent list and sibling episode lookups are fast.
/videos/电视剧/日韩剧/21世纪大君夫人 (2026)/Season 1/...S01E01.mp4/videos/电视剧/日韩剧/21世纪大君夫人 (2026)/Season 1after prewarm/videos/电视剧/国产剧/边水往事 (2024)/Season 1/...S01E01.mkv/videos/电视剧/国产剧/边水往事 (2024)/Season 1after prewarmEarlier source-mode validation
Directory list cache behavior:
/videos/电视剧refresh=true/videos/电视剧/videos/电视剧/videos/电视剧/欧美剧refresh=true/videos/电视剧/欧美剧/videos/电视剧/欧美剧/3体 (2024)refresh=true/videos/电视剧/欧美剧/3体 (2024)Search-index-assisted resolution validation:
/api/admin/index/progressis_done=true/api/fs/searchfor/ODS/tv17/.../S01E01/api/fs/get /videos/.../S01E01.mkv/api/fs/get /videos/.../S01E01.mkv/api/fs/get /ODS/tv17/.../S01E01.mkvRefresh behavior with Alias cache enabled:
/api/fs/list /videos/.../Season 1,refresh=false/api/fs/list /videos/.../Season 1,refresh=true/api/fs/list /videos/.../Season 1,refresh=falseLonger memory / latency comparison
Dataset: recursive list paths discovered under
/videos: 4074 directories.Cache-on measurement used a temporary 180-minute TTL during prewarm so entries would not expire before the 30-minute measurement. Local config was restored to 30 minutes after the test.
Notes:
go run, not the parentgo runprocess.alias_cache_max_entrieslimit.Checklist / 检查清单
/ 我已阅读 CONTRIBUTING。
/ 我确认此贡献符合仓库许可证、贡献规范和行为准则。
gofmt,go fmt, orprettierwhere applicable./ 我已按适用情况使用
gofmt、go fmt或prettier格式化变更代码。/ 我已在适用情况下请求相关维护者或代码所有者审查。
AI Disclosure / AI 使用声明
/ 此 PR 包含 AI 辅助内容。
Tools used / 使用工具:
Usage scope / 使用范围:
Code generation / 代码生成
Refactoring / 重构
Documentation / 文档
Tests / 测试
Translation / 翻译
Review assistance / 审查辅助
I have reviewed and validated all AI-assisted content included in this PR.
/ 我已审核并验证此 PR 中的所有 AI 辅助内容。
I have ensured that all AI-assisted commits include
Co-Authored-Byattribution./ 我已确保所有 AI 辅助提交都包含
Co-Authored-By归属信息。I can reproduce all AI-assisted content included in this PR without any AI tools.
/ 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。