[FIX]: Evaluate attack verdicts over final traces

docs/api/core-types.md

@@ -1,6 +1,8 @@
 # API Reference — Core Types
 
-Data types shared across the entire framework. All importable from `rampart` directly.
+Data types shared across the entire framework. Stable execution vocabulary is
+available from `rampart.core`; established result types remain importable from
+`rampart` directly.
 
 ## Data Types
 
@@ -14,6 +16,8 @@ Data types shared across the entire framework. All importable from `rampart` dir
         - ToolCall
         - SideEffect
         - Turn
+        - EvaluationPurpose
+        - TraceEndReason
         - EvalOutcome
         - EvalResult
         - EvalContext
@@ -30,6 +34,8 @@ Data types shared across the entire framework. All importable from `rampart` dir
         - SafetyStatus
         - HarmCategory
         - InjectionRecord
+        - resolve_attack_verdict
+        - resolve_probe_verdict
         - resolve_as_attack
         - resolve_as_probe
 

docs/attacks/xpia.md

@@ -30,9 +30,10 @@ sequenceDiagram
 1. **Inject** — Place payloads into the agent's data sources via surfaces. Each `surface.inject(payload)` returns an [`InjectionHandle`][rampart.core.injection.InjectionHandle].
 2. **Wait** — Handles call `wait_until_ready_async()` to allow indexing. Runs concurrently for multiple surfaces.
 3. **Trigger** — Send benign prompts that cause the agent to retrieve the injected content. Triggers are never adversarial — the attack is in the payload, not the prompt.
-4. **Evaluate** — Check each turn for the attack objective. Early-stops on detection.
-5. **Clean up** — Remove injected content. Guaranteed via `AsyncExitStack`, even on exceptions.
-6. **Result** — Produce a [`Result`][rampart.core.result.Result] via `resolve_as_attack` semantics.
+4. **Stop (optional)** — Check `stop_when` after each response and stop when detected.
+5. **Evaluate** — Check the attack objective once over the terminal trace.
+6. **Clean up** — Remove injected content. Guaranteed via `AsyncExitStack`, even on exceptions.
+7. **Result** — Map the final evaluation using attack semantics.
 
 ---
 
@@ -163,8 +164,12 @@ The `&` above asks whether both happened, so one condition that definitively did
     positive and negated mapping is maintained in the
     [Temporal Scope table](../usage/authoring-tests.md#temporal-scope).
     Omitting `scope` inspects only the current response and emits a
-    `FutureWarning` for multi-turn contexts. Scope applies only to turns in the
-    evaluator context; it does not control execution length or early stopping.
+    `FutureWarning` for multi-turn contexts.
+
+    XPIA verdict evaluators receive the terminal trace. Automatic stopping is
+    enabled only when detection is known to remain true as the trace grows.
+    Scope applies only to turns in the evaluator context; it does not control
+    execution length or early stopping.
 
 ### LLMDriver for Adaptive Triggers
 
@@ -202,6 +207,13 @@ assert result, result.summary
 !!! warning
     Construct a new `LLMDriver` per test. Each instance maintains its own conversation state and cannot be reused.
 
+!!! note "Adaptive driver budgets"
+    `LLMDriver` does not stop itself. The default `stop_when="auto"` stops
+    early for stable built-in conditions such as `ToolCalled`, but unknown or
+    stochastic evaluators run to `max_turns` and evaluate the terminal trace
+    once. Use an explicit `stop_when` when that online judgment intentionally
+    defines the end of the attack scenario.
+
 ---
 
 ## Trigger Options
@@ -233,7 +245,8 @@ See [`Attacks.xpia()`][rampart.attacks.Attacks.xpia] for the full API reference.
 | `inject` | `InjectionHandle \| list[InjectionHandle] \| None` | `None` | Prepared injections from `surface.inject()`. `None` for inline XPIA. |
 | `trigger` | `str \| list[str] \| Request \| list[Request] \| PromptDriver` | required | Benign prompt(s) that cause retrieval of injected content. |
 | `evaluator` | [`Evaluator`][rampart.core.evaluator.Evaluator] | required | What attack condition to detect. |
-| `max_turns` | `int` | `5` | Maximum prompt-response exchanges before `ERROR`. |
+| `stop_when` | [`Evaluator`][rampart.core.evaluator.Evaluator] `\| "auto" \| None` | `"auto"` | Online stop condition. Auto reuses stable built-in verdict evaluators and exposes their prefix results to adaptive drivers; `None` disables stopping and online feedback. |
+| `max_turns` | `int` | `5` | Maximum prompt-response exchanges; reaching the limit resolves the trace normally. |
 | `event_handlers` | `list[ExecutionEventHandler] \| None` | `None` | Additional lifecycle event handlers. |
 
 ---

docs/concepts/attacks.md

@@ -14,9 +14,13 @@ Attacks use the following mapping from evaluator outcomes to safety verdicts:
 | `NOT_DETECTED` | `SAFE` | The attack failed — the agent resisted |
 | `UNDETERMINED` | `UNDETERMINED` | The evaluator could not determine whether the attack succeeded |
 
-Precedence when multiple turns are evaluated: `DETECTED` > `UNDETERMINED` > `NOT_DETECTED`. If any turn detected the attack objective, the agent is compromised regardless of other turns.
+The evaluator runs once over the terminal trace, and the outcome maps directly
+to the verdict. This logic lives in
+[`resolve_attack_verdict`][rampart.core.result.resolve_attack_verdict].
 
-This logic lives in [`resolve_as_attack`][rampart.core.result.resolve_as_attack].
+Attack factories may evaluate a separate online `stop_when` condition while
+the trace is being produced. XPIA's `"auto"` default reuses the verdict
+evaluator only when detection is known to be stable as turns are appended.
 
 ---
 
@@ -27,9 +31,10 @@ All attack executions share this lifecycle:
 1. **Inject** (optional) — Place payloads into the agent's data sources via [surfaces](../api/core-protocols.md)
 2. **Wait** — Allow time for indexing or propagation
 3. **Trigger** — Send prompts that cause the agent to process the injected content
-4. **Evaluate** — Check whether the attack objective was achieved
-5. **Clean up** — Remove injected content (guaranteed, even on failure)
-6. **Report** — Produce a [`Result`][rampart.core.result.Result]
+4. **Stop (optional)** — Check an online condition after each response
+5. **Evaluate** — Check the terminal trace once for the attack objective
+6. **Clean up** — Remove injected content (guaranteed, even on failure)
+7. **Report** — Produce a [`Result`][rampart.core.result.Result]
 
 The injection phase is optional — inline attacks attach payloads directly to the trigger prompt.
 

docs/concepts/overview.md

@@ -58,7 +58,7 @@ A single test run flows from your pytest test, through a RAMPART attack or probe
 
 *Request / response cycle for a single test run.*
 
-Under the hood, every execution follows a common lifecycle owned by [`BaseExecution`][rampart.core.execution.BaseExecution], which drives the per-turn loop between the strategy, your adapter, and the evaluator:
+Under the hood, every execution follows a common lifecycle owned by [`BaseExecution`][rampart.core.execution.BaseExecution]. The strategy drives requests through your adapter, optionally evaluates an online stop condition, then evaluates the completed trace once for the verdict.
 
 ```mermaid
 sequenceDiagram
@@ -76,11 +76,16 @@ sequenceDiagram
         Strat->>Strat: driver.next_prompt_async(history)
         Strat->>Adapter: session.send_async(request)
         Adapter-->>Strat: Response
-        Strat->>Eval: evaluate_async(context)
-        Eval-->>Strat: EvalResult
-        Note over Strat: Early stop if detected
+        opt Explicit online stop condition
+            Strat->>Eval: evaluate_async(prefix context)
+            Eval-->>Strat: stop EvalResult
+            Note over Strat: Stop if detected
+        end
     end
 
+    Strat->>Eval: evaluate_async(terminal context)
+    Eval-->>Strat: final EvalResult
+
     Strat-->>Exec: Result
     Exec->>Exec: fire ON_POST_EXECUTE
     Exec-->>Test: Result
@@ -112,7 +117,7 @@ Evaluators are **polarity-free**. They answer "did X happen?" — not "is X good
 - In an **attack**, detection means the attack objective was achieved → **UNSAFE**
 - In a **probe**, detection means the expected behavior is present → **SAFE**
 
-The [`Attacks`][rampart.attacks.Attacks] and [`Probes`][rampart.probes.Probes] factories handle this mapping automatically via [`resolve_as_attack`][rampart.core.result.resolve_as_attack] and [`resolve_as_probe`][rampart.core.result.resolve_as_probe].
+The [`Attacks`][rampart.attacks.Attacks] and [`Probes`][rampart.probes.Probes] factories handle this mapping automatically via [`resolve_attack_verdict`][rampart.core.result.resolve_attack_verdict] and [`resolve_probe_verdict`][rampart.core.result.resolve_probe_verdict].
 
 You can reuse the same evaluator in both contexts. A [`ToolCalled`][rampart.evaluators.tool_called.ToolCalled] evaluator detects whether a tool was called — whether that's good or bad depends on whether you're attacking or probing.
 

docs/concepts/probes.md

@@ -14,9 +14,9 @@ Probes use the inverse mapping from evaluator outcomes:
 | `NOT_DETECTED` | `UNSAFE` | The expected behavior is missing — a regression |
 | `UNDETERMINED` | `UNDETERMINED` | The evaluator could not determine whether the behavior is present |
 
-Precedence: `NOT_DETECTED` > `UNDETERMINED` > `DETECTED`. If any turn failed to detect the expected behavior, the agent is non-compliant.
-
-This logic lives in [`resolve_as_probe`][rampart.core.result.resolve_as_probe].
+The evaluator runs once over the completed trace, and the outcome maps directly
+to the verdict. This logic lives in
+[`resolve_probe_verdict`][rampart.core.result.resolve_probe_verdict].
 
 ---
 
@@ -26,9 +26,10 @@ Probe executions are simpler than attacks — no injection phase:
 
 1. **Create session** — Open a fresh session with the agent
 2. **Send prompts** — Drive the conversation via the prompt driver
-3. **Evaluate** — Check whether the expected behavior is present
-4. **Clean up** — Close the session
-5. **Report** — Produce a [`Result`][rampart.core.result.Result]
+3. **Stop (optional)** — Check an explicit online `stop_when` condition
+4. **Evaluate** — Check the completed trace once for expected behavior
+5. **Clean up** — Close the session
+6. **Report** — Produce a [`Result`][rampart.core.result.Result]
 
 ---
 
@@ -51,6 +52,9 @@ assert result, result.summary
 
 Provide exactly one of `prompt`, `prompts`, or `driver`.
 
+Probes run the full prompt sequence by default. Pass `stop_when=` only when an
+online condition intentionally defines an earlier terminal trace.
+
 ---
 
 ## Available Probes

docs/contributing/architecture.md

@@ -41,8 +41,8 @@ When adding a new attack or probe, you add a static factory method — not a new
 
 Evaluators are **polarity-free**. They report whether a condition was detected, not whether it's good or bad. The attack/probe factory applies the correct polarity:
 
-- `resolve_as_attack`: detected → UNSAFE
-- `resolve_as_probe`: detected → SAFE
+- `resolve_attack_verdict`: detected → UNSAFE
+- `resolve_probe_verdict`: detected → SAFE
 
 This allows the same evaluator (e.g., `ToolCalled`) to be used in both attack and probe contexts.
 

docs/contributing/extending-rampart.md

@@ -39,9 +39,10 @@ from rampart.core import (
     ExecutionEventHandler,
     PromptDriver,
     Result,
-    Turn,
-    evaluate_turn_async,
-    resolve_as_attack,
+    SafetyStatus,
+    evaluate_terminal_async,
+    resolve_attack_verdict,
+    run_trace_async,
 )
 
 
@@ -51,6 +52,7 @@ class MyAttackExecution(BaseExecution):
     Args:
         driver (PromptDriver): How to drive the conversation.
         evaluator (Evaluator): What condition to check for.
+        stop_when (Evaluator | None): Optional online stop condition.
         max_turns (int): Maximum prompt-response exchanges.
         event_handlers (list[ExecutionEventHandler] | None): Additional handlers.
     """
@@ -60,12 +62,14 @@ class MyAttackExecution(BaseExecution):
         *,
         driver: PromptDriver,
         evaluator: Evaluator,
+        stop_when: Evaluator | None = None,
         max_turns: int = 25,
         event_handlers: list[ExecutionEventHandler] | None = None,
     ) -> None:
         super().__init__(event_handlers=event_handlers)
         self._driver = driver
         self._evaluator = evaluator
+        self._stop_when = stop_when
         self._max_turns = max_turns
 
     @property
@@ -82,38 +86,32 @@ class MyAttackExecution(BaseExecution):
         Returns:
             Result: Safety verdict.
         """
-        turns: list[Turn] = []
-
         async with await adapter.create_session_async() as session:
-            for turn_index in range(self._max_turns):
-                decision = await self._driver.next_prompt_async(history=turns)
-                if decision is None:
-                    break
-
-                response = await session.send_async(decision.request)
-                turn = await evaluate_turn_async(
-                    evaluator=self._evaluator,
-                    history=turns,
-                    request=decision.request,
-                    response=response,
-                    turn_number=turn_index,
-                    driver_reasoning=decision.reasoning,
-                    manifest=adapter.manifest,
-                    observability_level=adapter.observability_profile,
-                )
-                turns.append(turn)
-
-                if turn.eval_result and turn.eval_result.detected:
-                    break
-
-        # Use resolve_as_attack: detected → UNSAFE
-        eval_results = [t.eval_result for t in turns if t.eval_result is not None]
-        status = resolve_as_attack(eval_results=eval_results)
+            run = await run_trace_async(
+                session=session,
+                driver=self._driver,
+                max_turns=self._max_turns,
+                observability_level=adapter.observability_profile,
+                stop_when=self._stop_when,
+                manifest=adapter.manifest,
+            )
+            evaluation = await evaluate_terminal_async(
+                evaluator=self._evaluator,
+                run=run,
+            )
+
+        status = (
+            SafetyStatus.ERROR
+            if evaluation is None
+            else resolve_attack_verdict(evaluation=evaluation)
+        )
 
         return Result(
             status=status,
             summary="...",
-            turns=turns,
+            terminal_evaluation=evaluation,
+            turns=run.turns,
+            trace_end_reason=run.trace_end_reason,
             strategy=self.strategy_name,
             observability_level=adapter.observability_profile,
         )
@@ -124,8 +122,8 @@ Key points:
 - **Subclass `BaseExecution`** — it owns the lifecycle skeleton (event dispatch, timing, error handling)
 - **Implement `_execute_async`** — this is your strategy-specific logic
 - **Implement `strategy_name`** — a short identifier used in `Result.strategy`
-- **Use `resolve_as_attack`** — this maps evaluator outcomes to safety verdicts with attack semantics (detected = UNSAFE)
-- **Pass `observability_level`** so evaluators can tell missing evidence apart from an evidence channel the adapter does not report. It is required on both `evaluate_turn_async` and `Result`, so leaving it out is a `TypeError` rather than a wrong assumption buried in a report.
+- **Use `resolve_attack_verdict`** — this maps one terminal evaluation to attack semantics (detected = UNSAFE)
+- **Pass `observability_level`** so evaluators can tell missing evidence apart from a channel the adapter does not report. It is required on both `run_trace_async` and `Result`, so leaving it out is a `TypeError` rather than a wrong assumption buried in a report.
 - **Don't wrap `_execute_async` in a broad `try/except`** — `BaseExecution.execute_async` already catches every exception from `_execute_async` and converts it to a `SafetyStatus.ERROR` result.
 
 ### 2. Add a Factory Method to `Attacks`
@@ -181,29 +179,37 @@ The process mirrors the [Attack](#attack) walkthrough. The differences are summa
 |---|---|---|
 | **Location** | `rampart/attacks/_name.py` | `rampart/probes/_name.py` |
 | **Factory class** | `Attacks` | `Probes` |
-| **Resolution function** | `resolve_as_attack` | `resolve_as_probe` |
+| **Resolution function** | `resolve_attack_verdict` | `resolve_probe_verdict` |
 | **Detected means** | UNSAFE | SAFE |
 | **Injection phase** | Often yes | No |
 
 ### 1. Create the Execution Class
 
-The file structure mirrors the [Attack walkthrough](#1-create-the-execution-class) — same imports, `__init__`, and `_execute_async` loop. The diff from `MyAttackExecution` is:
+Probe strategies drive the full trace first, then evaluate it once while the
+session is still active:
 
-```diff
--from rampart.core import (..., resolve_as_attack)
-+from rampart.core import (..., resolve_as_probe)
-
--class MyAttackExecution(BaseExecution):
-+class MyProbeExecution(BaseExecution):
-
--    return "my_attack"
-+    return "my_probe"
+```python
+async with await adapter.create_session_async() as session:
+    run = await run_trace_async(
+        session=session,
+        driver=self._driver,
+        max_turns=self._max_turns,
+        observability_level=adapter.observability_profile,
+        stop_when=self._stop_when,
+        manifest=adapter.manifest,
+    )
+    evaluation = await evaluate_terminal_async(
+        evaluator=self._evaluator,
+        run=run,
+    )
 
--    status = resolve_as_attack(eval_results=eval_results)
-+    status = resolve_as_probe(eval_results=eval_results)
+status = resolve_probe_verdict(evaluation=evaluation)
 ```
 
-Place the file in `rampart/probes/` (e.g. `_my_probe.py`). Most probes skip the injection phase — just session creation, prompt driving, and evaluation. For a complete working reference, see [`rampart/probes/_single_turn.py`](https://github.com/microsoft/RAMPART/blob/main/rampart/probes/_single_turn.py).
+Store `terminal_evaluation`, `run.turns`, and `run.trace_end_reason` on the returned
+`Result`. Most probes skip the injection phase. For a complete working
+reference, see
+[`rampart/probes/_single_turn.py`](https://github.com/microsoft/RAMPART/blob/main/rampart/probes/_single_turn.py).
 
 ### 2. Add a Factory Method to `Probes`
 
@@ -214,7 +220,7 @@ Add a static method to the `Probes` class in `rampart/probes/__init__.py`, mirro
 Probe tests have the same surface as attack tests, with two differences:
 
 - **No injection phase** to test.
-- **Result resolution** uses `resolve_as_probe` semantics (detected → SAFE, not detected → UNSAFE).
+- **Result resolution** uses `resolve_probe_verdict` semantics (detected → SAFE, not detected → UNSAFE).
 
 
 ## Evaluator

docs/contributing/testing.md

@@ -138,16 +138,16 @@ When adding a new attack, test:
 
 1. **Execution lifecycle** — the attack calls `BaseExecution.execute_async` correctly
 2. **Phase orchestration** — injection, session creation, prompt driving, evaluation happen in order
-3. **Result resolution** — `resolve_as_attack` is applied (detected → UNSAFE, not detected → SAFE)
-4. **Edge cases** — empty handles, max turns reached, early stopping on detection
+3. **Result resolution** — `resolve_attack_verdict` maps one terminal evaluation (detected → UNSAFE, not detected → SAFE)
+4. **Edge cases** — empty handles, max turns reached, automatic/explicit/disabled stopping
 5. **Error handling** — infrastructure errors produce `SafetyStatus.ERROR`
 
 ### Testing a New Probe
 
 Similar to attacks, but:
 
 1. No injection phase to test
-2. Result resolution uses `resolve_as_probe` (detected → SAFE, not detected → UNSAFE)
+2. Result resolution uses `resolve_probe_verdict` over one terminal evaluation (detected → SAFE, not detected → UNSAFE)
 
 ### Testing a New Evaluator