Skip to content

[master] Fix 64433: Add dynamic loading of file_roots, pillar_roots, and thorium_roots - #64434

Open
bluesliverx wants to merge 11 commits into
saltstack:masterfrom
bluesliverx:dynamic-roots
Open

bluesliverx wants to merge 11 commits into
saltstack:masterfrom
bluesliverx:dynamic-roots

Conversation

@bluesliverx

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds dynamic expansion of file/pillar/thorium roots config.

What issues does this PR fix or reference?

Fixes: #64433

Previous Behavior

The roots were expanded only once at startup.

New Behavior

The roots are expanded on every access of the environments within the file_roots, pillar_roots, and thorium_roots options.

Merge requirements satisfied?

Commits signed with GPG?

No

@bluesliverx
bluesliverx requested a review from a team as a code owner June 6, 2023 23:22
@bluesliverx
bluesliverx requested review from twangboy and removed request for a team June 6, 2023 23:22
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 01:38 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 01:38 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 01:38 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 01:39 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 01:55 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 01:58 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 02:53 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 02:53 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 02:53 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 02:53 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 02:53 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 02:53 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 03:01 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 03:01 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 03:01 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 03:01 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 03:01 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 7, 2023 03:01 — with GitHub Actions Inactive
@anilsil anilsil added this to the Chlorine v3007.0 milestone Jun 7, 2023

@Ch3LL Ch3LL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get some additional test coverage that actually tests the full functionality. For example, adding a new directory path to file roots and being able to call that SLS file and same for pillar.

I'd like to also get some additional reviews on this.

Comment thread doc/ref/configuration/master.rst Outdated
Comment thread doc/ref/configuration/master.rst Outdated
Comment thread salt/config/__init__.py
@bluesliverx

Copy link
Copy Markdown
Contributor Author

@Ch3LL I have added tests and updated the documentation as requested. Let me know if you see anything else.

@bluesliverx
bluesliverx temporarily deployed to ci June 11, 2023 16:13 — with GitHub Actions Inactive
@bluesliverx
bluesliverx temporarily deployed to ci June 11, 2023 16:13 — with GitHub Actions Inactive
@bluesliverx

Copy link
Copy Markdown
Contributor Author

@twangboy, this is ready now I believe.

twangboy
twangboy previously approved these changes Mar 25, 2025
@bluesliverx

Copy link
Copy Markdown
Contributor Author

How does one go about getting this merged? :)

@twangboy

Copy link
Copy Markdown
Contributor

I'm thinking we'll get things fixed on 3006.x and 3007.x, then merge those forward into master. Then we can rebase and get this in.

@bluesliverx

Copy link
Copy Markdown
Contributor Author

@twangboy sorry to be annoying, any update on getting this merged in? Just curious.

@bluesliverx

Copy link
Copy Markdown
Contributor Author

@twangboy I am getting really annoying, I'm sure. But any help getting this merged in time for 3008 would be much appreciated so we can stop maintaining our own patch :)

@twangboy

Copy link
Copy Markdown
Contributor

When we get closer to release we will start merging pending PRs into master branch that are passing tests.

@twangboy

Copy link
Copy Markdown
Contributor

Please rebase this PR and fix conflicts

@bluesliverx

Copy link
Copy Markdown
Contributor Author

@twangboy done

twangboy
twangboy previously approved these changes Apr 1, 2026
Comment thread salt/utils/yamldumper.py Outdated
@dwoz

dwoz commented Apr 5, 2026

Copy link
Copy Markdown
Contributor

@bluesliverx needs rebase

@bluesliverx

Copy link
Copy Markdown
Contributor Author

@dwoz rebased

twangboy
twangboy previously approved these changes Jul 2, 2026
@twangboy

twangboy commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Please address the merge conflicts

@bluesliverx

Copy link
Copy Markdown
Contributor Author

@twangboy done

Comment thread salt/utils/dynamic_dict.py
Comment thread salt/utils/dynamic_dict.py
Comment thread salt/utils/dynamic_dict.py Outdated
… cache

Fixes 3 issues from PR review: items() wasn't overridden so callers
(e.g. salt/client/ssh) iterating file_roots.items() got raw unglobbed
paths; __deepcopy__ had a redundant overwrite that becomes a real
double-evaluation bug once items() is fixed; and glob expansion ran
unbounded on every access. Adds a per-entry TTL cache (default 5s),
exposed as the dynamic_roots_ttl config option for file_roots,
pillar_roots, and thorium_roots.
twangboy
twangboy previously approved these changes Jul 21, 2026
@bluesliverx

Copy link
Copy Markdown
Contributor Author

@twangboy this should be ready I believe, any chance of getting this merged in finally? Hopefully soon...

This branch was successfully deployed

1 active deployment
ci — d65831fa Deployed Sep 11, 2026 by twangboy via Build Onedir Packages / DEB (arm64) #27105
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE REQUEST] Support dynamic expansion of file_roots and pillar_roots

6 participants