Add --filter flag and @tag() decorator for selecting env vars#873
Merged
Conversation
Contributor
|
The changes in this PR will be included in the next version bump.
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
varlock-website | c4cf401 | Commit Preview URL Branch Preview URL |
Jul 14 2026, 10:57 PM |
commit: |
philmillman
approved these changes
Jul 9, 2026
philmillman
left a comment
philmillman
left a comment
Member
There was a problem hiding this comment.
lgtm, let's consider an in-depth tags and filters guide for the future
…nv vars Adds --filter to `varlock load`/`run` for selecting/omitting items by key name, glob, negation (!KEY), decorator (@sensitive/@required), or tag (#tagname). Tags are set via a new @tag() item decorator. The same selector language is also available as a filter= arg on @generate* code-generation decorators, so one schema can emit multiple generated files scoped to different subsets (e.g. a billing-only env.d.ts).
Spell out that non-negated selectors OR together regardless of kind (key/glob, @decorator, #tag), while negations subtract from that whole set rather than being scoped to a specific positive selector.
…ter bug Added tests for how key/glob, @decorator, and #tag selectors combine (union of positives, then subtract negatives, regardless of kind). The codegen-side mixed-kind tests caught a real bug: the `filter=` codegen arg was matching @sensitive/@required against bare ConfigItem getters, which aren't populated until resolveEnvValues() runs - but code generation runs before that. Fixed by filtering against TypeGenItemInfo (computed pre-resolution by getTypeGenInfo(), the same source codegen already uses for isSensitive/isRequired) instead, and widened the shared filter engine to a structural FilterableItem type so it can operate on either shape.
Regression coverage confirming @internal items stay excluded from getResolvedEnvObject/getSerializedGraph and codegen's collectTypeGenItems even when a filter would otherwise match them (e.g. by exact key name) - only --include-internal can opt an item back in.
Lets --filter be set without a CLI flag - useful for wrapper scripts, CI config, or build-time integrations that call varlock's loading APIs directly instead of going through the CLI. An explicit --filter flag always takes precedence, matching the existing _VARLOCK_REDACT_STDOUT precedent.
…n-full/agent exceptions
…longside this PR Removes the "known gap" caution about framework integrations not enforcing @internal exclusion - #874 fixes that at the CLI level (load --format json-full excludes internal items by default), so every integration inherits the guarantee with no code changes of their own. Both PRs are being merged together.
…ag selectors load/run previously always resolved (and validated) the entire graph regardless of --filter, so an unrelated invalid item outside the filter would still block a load - even though it wasn't going to be output anyway. A --filter using only key names/globs/tags is fully knowable from the schema before resolution, so we now resolve just those items plus their transitive dependencies, letting an unrelated broken var stay broken without blocking (e.g. scoping a build's validation to `--filter="#frontend"` independent of the backend vars). @sensitive/@required selectors can't be scoped this way - matches aren't knowable until the graph is already resolved - so those still fall back to resolving (and validating) everything, same as before.
theoephraim
force-pushed
the
add-filter-flag-and-tag-decorator
branch
from
0f270a6 to
be57739
Compare
…hen set ambiently An ambiently-set value for a filtered-out schema key passed straight through the process.env spread to the child (and, being excluded from the redaction map, printed unredacted) - contradicting the intent that a filtered-out var can't reach the child process. Strip excluded schema keys from the child env, mirroring the @internal handling, and drop them from the blob's override provenance metadata too. Also docs fixes: json-full does not show @internal items by default (needs --include-internal), em dash cleanup, codegen filter= cross-reference.
Replace the resolveItemFilterKeys/planFilterResolution pair (which each re-parsed the filter string, re-read _VARLOCK_FILTER, and duplicated the SchemaError->CliExitError wrapping) with a single parse-once getCliItemFilter() returning getResolveKeys/getFilterKeys. Bad filter syntax now errors before any loading work. Also precompile glob regexes per token instead of per item, and collapse codegen's unfiltered/filtered field caches into one map keyed by filter string. No behavior changes.
A typo'd key or tag silently produced empty output on load, or a child process with no schema vars on run, with exit 0. Warn on stderr (stdout stays clean for piped json/env/shell output); the command still succeeds.
Tag names must start with a letter or number, then letters, numbers, _, -, or . - rejecting the characters the filter language reserves (comma separator, !/@/# selector prefixes, whitespace, globs). Invalid names and empty @tag() calls are schema errors; #tag selectors in --filter/filter= enforce the same rule with a clear error instead of silently matching nothing. Duplicate tags now collapse.
Tag names are now letters/numbers/_/- only (still must start with a letter or number). Loosening later is backward-compatible; tightening would not be.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--filtertovarlock load/runfor selecting/omitting env vars by key name, glob (STRIPE_*), negation (!KEY), decorator (@sensitive/@required), or tag (#tagname)@tag()item decorator to attach tags to items, used by#tagnameselectors. Tag names are validated (start with a letter/number, then letters/numbers/_/-) so every tag stays selectable via#tagname; invalid names and empty@tag()calls are schema errors, and duplicates collapsefilter=arg to@generate*code-generation decorators (@generateTsTypes,@generatePythonEnv, etc.), so one schema can emit multiple generated files scoped to different subsets (e.g. a billing-onlyenv.d.ts)run, a filtered-out var can't reach the child process at all: excluded items are stripped from the__VARLOCK_ENVblob (including its override-provenance metadata) and removed from the child env even when set ambiently (same treatment as@internal). Without the ambient stripping,SECRET=x varlock run --filter='!@sensitive' -- ...passed the ambient value throughprocess.envand, since excluded items are left out of the redaction map, printed it unredacted_VARLOCK_FILTERenv var fallback for--filter, for wrapper scripts, CI config, or build-time integrations that don't go through the CLI (an explicit--filterflag still takes precedence)--filternow scopes resolution and validation too, not just output:load/runresolve only the selected items plus their transitive dependencies, so an unrelated broken item outside the filter no longer blocks the load - e.g. a build step scoped to--filter="#frontend"doesn't care if a backend-only var is misconfigured.@sensitive/@requiredselectors can't be scoped this way (matches aren't knowable until the graph is resolved), so those still resolve and validate everything, same as before.--filterthat matches no items (e.g. a typo'd tag) warns on stderr; the command still succeeds and stdout stays clean for piping@tag(), thefilter=arg on@generate*decorators, and@internal(pre-existing gap)Credit to @Jarva for #872, which proposed
--sensitive/--non-sensitiveflags onloadfor piping secrets to tools likefly secrets import. This PR generalizes that idea into a single--filterflag that also covers@required, tags, and key/glob selection.Fixed along the way: the codegen
filter=arg initially evaluated@sensitive/@requiredagainst rawConfigItemgetters, which aren't populated untilresolveEnvValues()runs - but code generation runs before that, so those selectors silently matched everything. Filtering now goes through the same pre-resolution computation (getTypeGenInfo()) codegen already uses forisSensitive/isRequired.Rebased on top of #874 (merged), which fixes
load --format json-fullto exclude@internalitems by default - discovered while documenting--filter's interaction with@internal.load.command.ts'sjson-fullbranch now combines both fixes (includeInternal+filterKeys); verified end-to-end that--include-internaland--filtercompose correctly together.Test plan
bun run --filter varlock typecheckbunx vitest runinpackages/varlock(1000 tests passing, including coverage for the filter parser,@tag()decorator, codegenfilter=wiring, mixed-kind selector interaction,@internalexclusion holding regardless of filter, the_VARLOCK_FILTERenv var/precedence, scoped resolution/validation for key/glob/tag filters, andgetSerializedGraphfilterKeys/override-provenance exclusion), plus smoke tests coveringrun --filterambient-env strippingbun run lint:fixbun run --filter varlock-website astro buildvarlock load --filter=...,varlock run --filter=...,varlock codegenwith mixedfilter="#tag,@sensitive",_VARLOCK_FILTER(plus flag precedence), scoped resolution (an unrelated invalid item outside a key/tag filter no longer fails the load, while--filter=@sensitivestill catches it), and--include-internal+--filtercomposed together end-to-end against real.env.schemafiles