Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 65 additions & 6 deletions docs/about-nemo-relay/concepts/scopes.mdx
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
---
title: "Scopes"
description: ""
description: "Understand scope hierarchy, lifetime, cleanup, and concurrent isolation."
position: 1
---
import { MermaidStyles } from "@/components/MermaidStyles";

{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0 */}

Expand All @@ -11,7 +13,7 @@ isolation.

## Why Scopes Exist

Scopes are the ownership backbone of NeMo Relay. Every tool call, LLM call, and
Scopes track where work belongs in NeMo Relay. Every tool call, LLM call, and
mark event attaches to a scope hierarchy.

That hierarchy lets the runtime:
Expand All @@ -30,14 +32,14 @@ A scope represents a logical unit of work such as:
- A request
- A workflow step
- A background task
- A nested function or tool orchestration boundary
- A nested function or tool workflow

Scopes are not just labels. They define ownership and visibility for other
runtime behavior.
Scopes are not just labels. They determine event parentage and which local
middleware and subscribers are visible.

## Scope Hierarchy and Ownership

Scopes form a tree. A child scope inherits the active execution context from its
Scopes form a tree. A child scope inherits the active context from its
parent and contributes new nested work beneath it.

That hierarchy determines:
Expand Down Expand Up @@ -90,6 +92,52 @@ and ends when it is popped or closed.
Scope-local middleware and subscribers are tied to the owning scope lifecycle.
When the scope closes, those registrations disappear automatically.

## Worked Scope Lifetime

Consider an agent that calls one tool and then one LLM:

1. The application or framework integration pushes an `Agent` scope beneath the
root and emits its start event.
2. The application registers middleware and a subscriber on the agent scope.
Both are visible while the agent or any of its nested scopes are active.
3. A managed tool wrapper emits the tool start event, runs the tool, and emits
the tool end event. The agent remains the active scope.
4. The application emits a mark under the active agent scope. The mark does not
change the stack.
5. A managed LLM wrapper emits the LLM start event, runs the model call, and
emits the LLM end event. The agent remains the active scope.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
6. The application or framework integration emits the agent end event and pops
the agent scope. Its local
middleware and subscriber are removed.

The active scope stack changes over time. The emitted events remain in a
parent-linked tree after their scopes close. A mark event attaches beneath the
active scope but does not push another scope onto the stack.

<MermaidStyles />

```mermaid
flowchart LR
subgraph Stack["Active scope stack over time"]
direction LR
s1["root → agent<br/>agent starts"] --> s2["root → agent<br/>tool lifecycle emits"]
s2 --> s3["root → agent<br/>mark emits"]
s3 --> s4["root → agent<br/>LLM lifecycle emits"]
s4 --> s5["root<br/>agent closes"]
end

subgraph Events["Emitted event tree"]
direction TB
root["root"] --> agent["agent start/end"]
agent --> tool["tool start/end"]
agent --> mark["mark"]
agent --> llm["llm start/end"]
end
Comment thread
afourniernv marked this conversation as resolved.
```

Agent-local middleware and subscribers are visible during the first four stack
states. They are no longer visible after the agent scope closes.

## Semantic Payloads

Scopes may expose semantic `input` and `output` payloads on their emitted start
Expand All @@ -112,6 +160,17 @@ itself.
Context isolation keeps concurrent requests, tenants, and agents from sharing scope-
local state accidentally.

Choose the context behavior based on whether the work belongs to the same
logical trace and whether it runs concurrently:

| Work | Context Behavior |
| --- | --- |
| Sequential nested work in the same trace | Reuse the active stack |
| Concurrent branches in the same trace | Fork the stack for each branch |
| Independent work | Start with a fresh isolated stack |
| Same-trace work crossing a process boundary | Carry Relay propagation context; scope-local registrations do not cross the boundary |
| Independent work crossing a process boundary | Start with a fresh isolated stack instead of importing context |

### Why Isolation Matters

Concurrent requests must not share the same active scope stack accidentally.
Expand Down
Loading