feat(sdk): add route policy enforcement to CompositeBackend and filesystem middleware#2511
feat(sdk): add route policy enforcement to CompositeBackend and filesystem middleware#2511Nick Hollon (nick-hollon-lc) wants to merge 14 commits intomainfrom
Conversation
a8b93cc to
991bb66
Compare
There was a problem hiding this comment.
Security Issues
-
Authorization Bypass in RoutePolicy Enforcement (grep/agrep, glob/aglob)
In CompositeBackend, the default_policy is not enforced for the default backend in certain code paths. Specifically, when a path does not match any route, grep/agrep fall back to calling the default backend without checking _policy_error(). Similarly, in glob/aglob, the aggregation path enforces policies for routes but not for the default backend. An attacker can bypass intended restrictions (e.g., disallowed grep/glob on the default backend) by specifying a non-routed path. -
Authorization Bypass in RoutePolicy Enforcement (ls/als aggregate case)
In CompositeBackend.ls()/als() the policy is only enforced when a path matches a route. In the "default handling" branch (aggregate of default + each route root), there is no _policy_error() check applied to the default backend nor to the per-route entries. If a route or default_policy is intended to block ls, an attacker can call ls("/") to enumerate entries from restricted paths, bypassing the policy.
Recommendations
- Before invoking default backend operations in grep/agrep fallbacks and in glob/aglob aggregation, enforce default_policy via _policy_error(). If blocked, return an error (or skip aggregation) rather than executing the call.
- In ls()/als() aggregate branch, call _policy_error("ls", ...) for the default backend and for each route before including results. If disallowed, return an error (default) or skip that route (routes).
e0e96fb to
70d21f0
Compare
…icies When a CompositeBackend has policies that make a backend method impossible on every route and the default, the corresponding tools are now removed from the LLM's tool list in wrap_model_call/awrap_model_call. This avoids wasted model turns where the LLM invokes a tool that can only return a policy error. - Add `globally_blocked_methods()` to `CompositeBackend` - Extract `_filter_tools_by_backend()` and `_build_and_attach_system_prompt()` from duplicated wrap/awrap logic in `FilesystemMiddleware` - Filter write_file/edit_file/execute (and others) when blocked everywhere - Also catches execute blocked by policy despite SandboxBackendProtocol support
70d21f0 to
de9f146
Compare
Sydney Runkle (sydney-runkle)
left a comment
There was a problem hiding this comment.
first pass review, i'm very excited about this feature!
what's the actual use case for removing tools? i presume we almost always want the agent to be able to write to artifacts / scratch files, so my hunch is that this might be over-engineering. we could always go add this back later, prompting might be a good enough solution for now?
| # Only entries where the names diverge are listed; tools whose name | ||
| # matches the backend method exactly (ls, glob, grep, execute) are | ||
| # handled by the identity fallback in _tool_name_to_method(). | ||
| _TOOL_TO_METHOD: dict[str, str] = { |
There was a problem hiding this comment.
maybe cleaner if we just have a full map here?
| # Add route policy descriptions if configured | ||
| policy_prompt = _build_policy_prompt(resolved_backend) | ||
| if policy_prompt: | ||
| prompt_parts.append(policy_prompt) | ||
|
|
There was a problem hiding this comment.
we should sync w/ vivek (@vtrivedy) on this addition (since it's a prompt change), he's our harness guru
| update tagging a newly evicted message. | ||
| """ | ||
| resolved_backend = self._get_backend(request.runtime) # ty: ignore[invalid-argument-type] | ||
| request, has_execute_tool, backend_supports_execution = self._filter_tools_by_backend(request, resolved_backend) |
There was a problem hiding this comment.
should this helper just be called in _build_and_attach_system_prompt? kind of hard to follow this logic as a first time reader
There was a problem hiding this comment.
this one i'm not sure about so maybe we can chat? pushing it into _build_and_attach_system_prompt would mean that we modify the tools and the system prompt on the request in one function but then later modify the messages. i kind of like keeping it all top level in this function so we have modify tools, modify system prompt, modify messages. that being said maybe we should put the result of _filter_tool_by_backend into an object so we aren't doing the tuple unpacking?
| """ | ||
| return method in self.allowed_methods | ||
|
|
||
| def describe(self) -> str: |
There was a problem hiding this comment.
should this just be a description @property?
|
|
||
| allowed_methods: set[str] | ||
|
|
||
| def is_allowed(self, method: str, **context: Any) -> bool: # noqa: ARG002 |
There was a problem hiding this comment.
is this necessary? what is **context?
There was a problem hiding this comment.
it sorta was more forward looking. if you needed to pass additional context on a new route policy type to properly implement is_allowed you could use this. but also practically speaking we don't really need it now since none of the callers in CompositeBackend pass anything thru so it would have to be a fully custom implementation that just happened to use our RoutePolicy
There was a problem hiding this comment.
yeah let's go minimal here, no reason to expose if we don't have a use for it yet
…lso adjust system prompt to state the allowed tools for each route instead of the allowed methods
Sydney Runkle (sydney-runkle)
left a comment
There was a problem hiding this comment.
generally excited about this! PR could use a bit of cleanup
i think too much indirection w/ helper methods makes the code a bit hard to digest
i think claude might be able to help w/ refactor here
let's figure out what our plan is for default policy + adding the delete tool. if you have any questions there you can ask caspar who made the delete feature request
| bare-backend routes (without an explicit `Route.policy`). | ||
| Does **not** override explicit `Route.policy` values. | ||
| artifacts_root: Root path for artifacts, such as messages offloaded | ||
| by middleware. Defaults to `"/"`. |
There was a problem hiding this comment.
what's w/ this?
| self.default_policy = default_policy | ||
|
|
||
| backend_routes: dict[str, BackendProtocol] = {} | ||
| self._policies: dict[str, RoutePolicy | None] = {} |
There was a problem hiding this comment.
inconsistent private vs public, let's decide what makes sense
| self._policies[prefix] = None | ||
|
|
||
| self.routes = backend_routes | ||
| self.sorted_routes = sorted(backend_routes.items(), key=lambda x: len(x[0]), reverse=True) |
There was a problem hiding this comment.
what is this doing 😆 ik we had before but i'm confused
| return GrepResult(matches=raw) | ||
|
|
||
| def grep( | ||
| def grep( # noqa: C901, PLR0911 |
There was a problem hiding this comment.
what's up w/ this
| else: | ||
| backend_batches[backend].append((idx, stripped_path, content)) | ||
|
|
||
| # Process each backend's batch |
There was a problem hiding this comment.
why are we removing? let's be more careful
…mpt with route policies
…s that got stomped
… BackendProtocol to be opt in
| ) | ||
|
|
||
| __all__ = [ | ||
| "DEFAULT_BACKEND_METHODS", |
There was a problem hiding this comment.
does this need to be in the public API?
|
|
||
| # Sort routes by length (longest first) for correct prefix matching | ||
| self.sorted_routes = sorted(routes.items(), key=lambda x: len(x[0]), reverse=True) | ||
| def policy_for_route(self, route_prefix: str | None) -> RoutePolicy | None: |
There was a problem hiding this comment.
does this need to be in public API? (e.g., should we do _policy_for_route)
|
|
||
| if backend.default_policy is not None: | ||
| lines.append( | ||
| f"- Default policy (all other paths): allowed tools: {_allowed_tools(backend.default_policy, supports_execute=supports_execute)}" |
There was a problem hiding this comment.
let's make sure this makes sense if we only have the default policy
| message is tagged in state via `ExtendedModelResponse`. | ||
| resolved_backend: BackendProtocol, | ||
| *, | ||
| has_execute_tool: bool, |
There was a problem hiding this comment.
is this in the scope of the PR?
| self, | ||
| default: BackendProtocol | StateBackend, | ||
| routes: dict[str, BackendProtocol], | ||
| routes: dict[str, Route | BackendProtocol], |
There was a problem hiding this comment.
maybe question for design doc but should routes remain a required arg? should I be able to create CompositeBackend(default=backend, default_policy=...)?
|
abandoning this in favor of #2633 |
Adds
RoutePolicyandRoutedataclasses toCompositeBackend, enabling declarative access control on routes. Policies restrict which backend methods (e.g.,read,write,edit,execute) are allowed on a given route or on the default backend. TheFilesystemMiddlewaresystem prompt is updated to describe active policies so the model can plan accordingly.Design
RoutePolicy— dataclass withallowed_methods: set[str], an overridableis_allowed(method, **context)for future programmatic policies (e.g., user-based access control), and adescribe()method for system prompt rendering.Route— couples aBackendProtocolwith an optionalRoutePolicy.CompositeBackendacceptsdict[str, Route | BackendProtocol](backwards compatible) and an optionaldefault_policythat applies to both bare-backend routes and the default backend.Route.policyoverridesdefault_policy. No policy = no restrictions.CompositeBackend(single chokepoint), not individual backends.FilesystemMiddlewareappends a## Route Policiessection to the system prompt when policies are configured.