Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,14 @@
prefix and an id already spelled for Bedrock as written; server-side refusal fallbacks are never sent there (Bedrock offers
none); a call whose credential chain is empty is a `503 AWS credential missing` by name. `llm.health` reports `region`
beside `backend` and `model`, and its check resolves the credential chain the way a call would, without spending a token.
The routes and the contract are unchanged: the shared vector file gains three Bedrock cases (66), byte-identical with the
Node pack, and the Bedrock client, its region and credential checks are pinned by unit tests. **Upgrade note:** none - the
Bedrock serves Claude on two surfaces and an account may be enabled for one only: `llm.aws.api` selects `mantle` (the
default, the Anthropic-shaped endpoint with the prefixed ids) or `runtime` (the classic one, every model id sent as written
- an inference profile id as the account lists it), and `llm.health` names the surface. Certified live on 2026-10-14 against
an enterprise account, negative controls first (a wrong secret a 401 by name, no credential a 503 by name), then a chat
reply, a structured verdict and a progressive stream on Haiku 4.5 through the runtime surface, from both packs and through
the Java engine's shipped triage graph and streaming relay (the engines' `docs/test-reports/llm-helper-certification.md`, Scenario 8, and this pack's own report).
The routes and the contract are unchanged: the shared vector file gains five Bedrock cases (68), byte-identical with the
Node pack, and the Bedrock clients, their region and credential checks are pinned by unit tests. **Upgrade note:** none - the
Claude API backend is the default and behaves as before; `llm.backend` now also accepts `bedrock`.

## Version 4.12.21, 10/7/2026
Expand Down
32 changes: 32 additions & 0 deletions docs/test-reports/llm-helper-certification.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,38 @@ nothing is gathered or sent once. The long gaps are the API's own pacing, the sa
no credential gave 503 through Java on all three layers and through Rust on the graph, and an invalid key gave Anthropic's 401 with its request id when this pack's client called the helper directly. The same message came from both helpers, through every layer. All traces that touched this helper rebuild
as one connected tree.

## Verified live on AWS Bedrock (2026-10-14)

The helper's second backend (`llm.backend: 'bedrock'`), driven against an enterprise AWS account in `us-east-1` with temporary IAM
credentials - a key pair and a session token - in the helper's environment and no vendor key anywhere. Negative controls first.

| Step | Result |
|---|---|
| A corrupted secret | `401 LLM provider error - 401 authentication_error: The request signature we calculated does not match ...` |
| No credential at all | health `503 AWS credential missing`; the call a bare 500 until this pack mapped the SDK's wording (below) |
| `llm.chat`, Haiku 4.5 | `200`, `ready`, 14 input / 4 output tokens |
| `llm.chat` with a schema | `{"verdict": "bug", "confidence": 0.95}`, 218 / 17 |
| `llm.stream` | 4 batches, the first after 815 ms, `eof` with usage 20 / 23 |
| Java engine, the shipped `support-triage` graph | `bug` → `bug-filed`, 240 / 29 |
| Java engine, the streaming relay | 4 `data` frames, then `done` with usage, the trace id and the correlation id |

Every call ran on `us.anthropic.claude-haiku-4-5-20251001-v1:0` through the classic runtime surface (`llm.aws.api: 'runtime'`): the
account was enabled for that surface and not for the Anthropic-shaped one the SDK prefers, and for Haiku 4.5 and not for the newer
model generation. The full account of the surfaces and the gates is Finding 7 of the
[engine report](https://accenture.github.io/mercury-composable/test-reports/llm-helper-certification/).

## What the Python SDK needed on Bedrock

- **The Bedrock clients live at `anthropic.lib.bedrock`** (`AsyncAnthropicBedrockMantle`, `AsyncAnthropicBedrock`); the top level
re-exports them through a star import that the type checker reads as private, so the helper imports the package path.
- **An empty credential chain is a bare `RuntimeError`** raised while signing - `Could not resolve AWS credentials from session`
from the Mantle client, `could not resolve credentials from session` from the classic one - so the helper matches the words both
wordings carry, without case, and maps it to `503 AWS credential missing`. The health check resolves a boto3 session's
credentials the way a call would, so a probe tells a missing credential from a present one without spending a token.
- **No region is a construction error**, an `AnthropicError` from the Mantle client and a `ValueError` from the classic one; both
are the helper's `503 AWS region missing`, raised before any call.
- **The reply carries no request id:** Bedrock sends no `request-id` header the helper reads.

## What the Python SDK needed

- **No credential is not an API error.** `anthropic` 1.x raises a bare `TypeError` ("Could not resolve authentication method…")
Expand Down
32 changes: 26 additions & 6 deletions examples/llm-helper/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,15 @@ export AWS_REGION=us-east-1 # or llm.aws.region; the AWS credenti
mercury-serve examples/llm-helper/llm_helper.py -Dllm.backend=bedrock
```

If the account is enabled for Bedrock's classic runtime and not for its Anthropic-shaped endpoint (the default
surface answers `403 ... not available for this account` on a model the console's playground runs), switch the
surface and name the model as an inference profile id, exactly as the account lists it:

```bash
mercury-serve examples/llm-helper/llm_helper.py -Dllm.backend=bedrock -Dllm.aws.api=runtime \
-Dllm.model=us.anthropic.claude-haiku-4-5-20251001-v1:0
```

The app listens on port 8086 (`rest.server.port` in `resources/application.yml`). Map its routes
from the engine's `event-over-http.yaml`, as the Java and Rust MiniGraph Playgrounds do:

Expand All @@ -54,7 +63,7 @@ token batches as they are produced.
| `prompt` or `messages` | A single user turn, or conversation turns `[{role: user\|assistant, content}]`. `messages` wins when both are given. Content is plain text. |
| `system` | Optional system prompt (text). |
| `schema` | `llm.chat` only. A JSON schema; the reply is then structured output, returned parsed as `data`. `additionalProperties` defaults to `false`. A schema on `llm.stream` is a 400. |
| `params.model` | The model. Default `claude-opus-5-5` (`llm.model`). On Bedrock a bare id is sent with the `anthropic.` prefix; an id already spelled for Bedrock is sent as written. |
| `params.model` | The model. Default `claude-opus-5-5` (`llm.model`). On Bedrock's default surface a bare id is sent with the `anthropic.` prefix and an id already spelled for Bedrock as written; on the classic runtime surface every id is sent as written, an inference profile id as the account lists it. |
| `params.max_tokens` | Output cap. Default 16000 (`llm.max.tokens`). |
| `params.timeout_ms` | `llm.chat`: a deadline for the whole call, SDK retries included. `llm.stream`: the idle allowance between events. Default 60000 (`llm.timeout.ms`). |
| `params.effort` | `low`, `medium`, `high`, `xhigh` or `max`. Sent only when set; otherwise the model's own default applies. A model that takes no effort parameter answers 400. |
Expand Down Expand Up @@ -121,6 +130,7 @@ the 422 above, not an empty stream. Raise `params.max_tokens` or lower `params.e
|---|---|---|
| `llm.backend` | `anthropic` | The route to the models: `anthropic` (the Claude API) or `bedrock` (AWS Bedrock); see Backends. |
| `llm.aws.region` | unset | The AWS region a Bedrock call is signed for. Unset, the SDK reads `AWS_REGION` / `AWS_DEFAULT_REGION`. |
| `llm.aws.api` | `mantle` | The Bedrock surface: `mantle`, the Anthropic-shaped endpoint, or `runtime`, the classic one; see Backends. |
| `llm.model` | `claude-opus-5-5` | The default model. |
| `llm.max.tokens` | `16000` | The default output cap. |
| `llm.timeout.ms` | `60000` | The default deadline (chat) or idle allowance (stream). |
Expand Down Expand Up @@ -166,7 +176,7 @@ contract and the vector file are the same on every backend.
| `llm.backend` | Client | Credential | Model id | Server-side fallbacks |
|---|---|---|---|---|
| `anthropic` (default) | the Claude API, `AsyncAnthropic` | `ANTHROPIC_API_KEY`, or whatever the SDK resolves | as written | on the models that support them |
| `bedrock` | AWS Bedrock, the SDK's `AsyncAnthropicBedrockMantle` | the AWS credential chain through IAM (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, a profile, a role), or a Bedrock API key in `AWS_BEARER_TOKEN_BEDROCK` | a bare id gets the `anthropic.` prefix; an id already spelled for Bedrock is sent as written | none - the request never carries them |
| `bedrock` | AWS Bedrock, the SDK's `AsyncAnthropicBedrockMantle` (`llm.aws.api: mantle`) or `AsyncAnthropicBedrock` (`runtime`) | the AWS credential chain through IAM (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, a profile, a role), or a Bedrock API key in `AWS_BEARER_TOKEN_BEDROCK` | `mantle`: a bare id gets the `anthropic.` prefix, an id already spelled for Bedrock is sent as written; `runtime`: every id as written | none - the request never carries them |

**AWS Bedrock.** Install the extra (`pip install 'mercury-composable[llm,bedrock]'` - it adds boto3,
which signs each call), set `llm.backend: 'bedrock'` and a region (`llm.aws.region`, or `AWS_REGION`
Expand All @@ -180,6 +190,16 @@ Effort, structured output and streaming work as on the Claude API. The SDK's cli
fallback middleware is not wired in; a refusal on Bedrock is the 422 or the `stop_details` of the
contract, like a refusal on the Claude API with `llm.fallbacks: off`.

**Two surfaces.** Bedrock serves Claude on two endpoints, and an account may be enabled for one and not the
other. `llm.aws.api: 'mantle'` (the default) is the Anthropic-shaped endpoint the SDK prefers, which takes the
vendor-prefixed ids above. `llm.aws.api: 'runtime'` is the classic Bedrock runtime, which takes an inference profile
id exactly as the account lists it (`us.anthropic.claude-haiku-4-5-20251001-v1:0` - they carry dates and versions no
rule can derive, so the helper sends every id as written there). The sign that an account lacks the default surface
is a `403 ... is not available for this account` on a model the console's playground runs: switch the surface and
name the model as a profile id. The same message on both surfaces, with the model agreement in place, is an account
enablement for that model generation - a conversation with the AWS account team, not a setting. `llm.health` reports
`api` beside `backend`, `model` and `region`. Bedrock sends no `request_id` the helper reads, so a reply carries none.

## What it does not do

No tools or function calling, no images or documents, no `system` turns inside `messages`, no
Expand All @@ -189,10 +209,10 @@ deliberate: the helper is the bounded AI node of a graph, not an agent runtime.
## Tests

`tests/test_llm_helper.py` runs the shared `tests/vectors/llm-helper-vectors.json` against a fake of
the SDK (66 cases: request validation, the exact SDK call, replies, the error contract, streaming,
the shape of a Bedrock call), plus the tests a vector cannot express (token batches are never held
back, no prompt text in a log, the deadline cancels the call, trace annotations, health, the backend
seam, the Bedrock client with its region and credential checks). No token is spent and no credential
the SDK (68 cases: request validation, the exact SDK call, replies, the error contract, streaming,
the shape of a Bedrock call on either surface), plus the tests a vector cannot express (token batches are
never held back, no prompt text in a log, the deadline cancels the call, trace annotations, health, the
backend seam, the Bedrock clients with their region and credential checks). No token is spent and no credential
is needed.

```bash
Expand Down
Loading
Loading