Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -582,9 +582,19 @@ LLM-invoked tools. Per-tool `input_policy` (trusted parameter injection) is
still not implemented — see §11.

**Model providers (✅ done, not in the original task list):**
`agent_engine/models/factory.py` builds chat models via `init_chat_model` for
both **Anthropic** and **Amazon Bedrock** (`ChatBedrockConverse`), with clear
configuration errors for missing settings.
`agent_engine/models/factory.py` builds chat models for **Anthropic** (via
`init_chat_model`), **Amazon Bedrock** (`ChatBedrockConverse`), **Google
Gemini** (`ChatGoogleGenerativeAI`), and **OpenAI** (`ChatOpenAI`), with clear
configuration errors for missing settings. The `openai` provider is also the
entry point for any OpenAI-compatible endpoint: `agent_engine/models/presets.py`
holds a small registry mapping known vendor ids (`zai`, `deepseek`, `moonshot`,
`groq`, `xai`, `openrouter`) to their `base_url`/`api_key_env`, so `provider:
zai` in YAML resolves both without either field being typed out. `base_url`
and `api_key_env` remain available directly on `provider: openai` for any
vendor not in the registry, a self-hosted server (Ollama, vLLM), or to
override a listed vendor's preset (routing through an internal proxy, say).
No new dependency either way, it's all `ChatOpenAI`. See
[YAML_SPEC.md](YAML_SPEC.md#any-openai-compatible-endpoint).

**CLI (0008 — ✅ done):**
`agentctl validate`, `agentctl inspect` (offline summary: agents, MCPs, hooks,
Expand Down
3 changes: 2 additions & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ table above:
| ---------- | ------ | ----- |
| Runtime hooks (auth/policy/audit/context-enrichment, incl. `transform_tool_result`) | ✅ done | `agent_engine/runtime/hooks/`, [RUNTIME_HOOKS.md](RUNTIME_HOOKS.md) |
| Per-run execution-limit guardrails | ✅ done | `agent_engine/core/execution.py`, `agent_engine/runtime/execution.py`, [EXECUTION_LIMITS.md](EXECUTION_LIMITS.md) |
| Amazon Bedrock model provider (in addition to Anthropic) | ✅ done | `agent_engine/models/factory.py` |
| Amazon Bedrock, Google Gemini, and OpenAI model providers (in addition to Anthropic) | ✅ done | `agent_engine/models/factory.py` |
| Any OpenAI-compatible endpoint, with `provider: zai`/`deepseek`/`moonshot`/`groq`/`xai`/`openrouter` shorthand and a `base_url`/`api_key_env` escape hatch for anything else (self-hosted Ollama/vLLM, an unlisted vendor, an internal proxy) | ✅ done | `agent_engine/models/factory.py`, `agent_engine/models/presets.py`, [YAML_SPEC.md](YAML_SPEC.md#any-openai-compatible-endpoint) |
| Conversation persistence (SQLite default, sessions, users) | ✅ done | `agent_manager/` |
| Embeddable JS/React chat widget | ✅ done | `agent_manager/api/static/widget/` |
| Long-term / cross-conversation memory | ⏳ planned | — |
Expand Down
56 changes: 56 additions & 0 deletions docs/YAML_SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,62 @@ For OpenAI, set `OPENAI_API_KEY` in your environment and install the provider
extra with `pip install "agent-engine[openai]"`. Any OpenAI model your key can
access may be used via `name`. Secrets must never be stored in YAML.

### Any OpenAI-compatible endpoint

The `openai` provider is not limited to `api.openai.com`. A handful of
well-known OpenAI-compatible vendors are wired in as `provider` shorthand, so
`base_url` and `api_key_env` resolve automatically:

```yaml
model:
provider: zai
name: glm-5.2
temperature: 0.2
```

| `provider` | Vendor | `base_url` | `api_key_env` |
| ---------- | ---------- | ---------------------------------------- | -------------------- |
| `zai` | Z.AI (GLM) | `https://api.z.ai/api/coding/paas/v4` | `ZAI_API_KEY` |
| `deepseek` | DeepSeek | `https://api.deepseek.com/v1` | `DEEPSEEK_API_KEY` |
| `moonshot` | Moonshot | `https://api.moonshot.ai/v1` | `MOONSHOT_API_KEY` |
| `groq` | Groq | `https://api.groq.com/openai/v1` | `GROQ_API_KEY` |
| `xai` | xAI (Grok) | `https://api.x.ai/v1` | `XAI_API_KEY` |
| `openrouter` | OpenRouter | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` |

`name` must be a model ID that vendor's endpoint actually serves (`glm-5.2`
for Z.AI, `deepseek-chat` for DeepSeek, and so on); it is passed through
unmodified, no provider picks a default model on your behalf.

For anything not in the table (a different vendor, a self-hosted Ollama or
vLLM server, a proxy in front of one of the listed vendors), stay on
`provider: openai` and set `base_url` and `api_key_env` directly:

```yaml
model:
provider: openai
name: llama3.1
base_url: http://localhost:11434/v1
api_key_env: OLLAMA_API_KEY
```

`api_key_env` names the environment variable holding the key, it is never
the key itself, and secrets must never be stored in YAML. If a local server
doesn't check the key at all, point `api_key_env` at any variable set to a
placeholder value; the field is still required so the request always sends
an `Authorization` header.

`base_url`/`api_key_env` also override a listed vendor's preset when both a
`provider` shorthand and one of these fields are set, useful for routing a
known vendor through an internal proxy without giving up the shorthand:

```yaml
model:
provider: zai
name: glm-5.2
base_url: https://llm-proxy.internal.example.com/zai
api_key_env: INTERNAL_PROXY_KEY
```

---

## Graph Topology
Expand Down
2 changes: 2 additions & 0 deletions src/agent_engine/core/spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ class ModelConfig:
region: str | None = None
max_tokens: int | None = None
top_p: float | None = None
base_url: str | None = None
api_key_env: str | None = None


@dataclass(frozen=True)
Expand Down
2 changes: 2 additions & 0 deletions src/agent_engine/engine/langgraph/engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,8 @@ def _model_factory_kwargs(factory: ModelFactory, model: NodeModelConfig) -> dict
"region": model.region,
"max_tokens": model.max_tokens,
"top_p": model.top_p,
"base_url": model.base_url,
"api_key_env": model.api_key_env,
}
present: dict[str, object] = {
key: value for key, value in optional.items() if value is not None
Expand Down
48 changes: 39 additions & 9 deletions src/agent_engine/models/factory.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
from langchain_core.language_models import BaseChatModel
Comment thread
AmitAvital1 marked this conversation as resolved.

from agent_engine.logging_config import log
from agent_engine.models.presets import OPENAI_COMPAT_PRESETS

logger = logging.getLogger(__name__)

Expand All @@ -34,6 +35,8 @@ def build_chat_model(
region: str | None = None,
max_tokens: int | None = None,
top_p: float | None = None,
base_url: str | None = None,
api_key_env: str | None = None,
) -> BaseChatModel:
"""Construct a chat model from flat config fields.

Expand All @@ -44,11 +47,25 @@ def build_chat_model(
if not model_name:
raise ModelConfigurationError("Model name must not be empty.")

# A preset id (zai, deepseek, moonshot, groq, xai, openrouter, ...) is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about checking if provider is gemini, but user declare base url or api key, or just one of them? we should error him on validate yaml that its not acceptable

# shorthand for provider: openai with that vendor's base_url/api_key_env
# filled in. YAML-supplied base_url/api_key_env still win if both are
# set explicitly alongside a preset id, so a preset can be overridden
# (a proxy in front of it, a different key variable name) without
# switching away from the shorthand.
preset_id = normalized_provider
preset = OPENAI_COMPAT_PRESETS.get(preset_id)
if preset is not None:
base_url = base_url or preset.base_url
api_key_env = api_key_env or preset.api_key_env
normalized_provider = "openai"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we remove preset always provider will be openai

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, right, drop the registry and this branch goes with it, proider is just openai plus whatever base_url/key the user passed. they live or die together


log(
logger,
logging.INFO,
"llm configured",
provider=normalized_provider,
provider_preset=preset_id if preset is not None else None,
model=model_name,
temperature=temperature,
region=region,
Expand Down Expand Up @@ -81,8 +98,10 @@ def build_chat_model(
temperature=temperature,
max_tokens=max_tokens,
top_p=top_p,
base_url=base_url,
api_key_env=api_key_env,
)
supported = ", ".join(_SUPPORTED_PROVIDERS)
supported = ", ".join((*_SUPPORTED_PROVIDERS, *OPENAI_COMPAT_PRESETS))
raise ModelConfigurationError(
f"Unsupported model provider '{provider}'. Supported providers: {supported}."
)
Expand Down Expand Up @@ -205,11 +224,19 @@ def _build_openai_model(
temperature: float | None,
max_tokens: int | None,
top_p: float | None,
base_url: str | None = None,
api_key_env: str | None = None,
) -> BaseChatModel:
api_key = _resolve_openai_api_key()
# `api_key_env` lets this provider point at ANY OpenAI-compatible chat
# completions endpoint (Z.AI, DeepSeek, Moonshot, Groq, xAI, OpenRouter,
# a local Ollama/vLLM server, ...) by naming the env var that holds that
# vendor's key. Defaults to OPENAI_API_KEY to keep existing YAML working
# unchanged.
key_env_var = api_key_env.strip() if api_key_env and api_key_env.strip() else "OPENAI_API_KEY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If api_key_env is "" we not should error so user meant to override and not use OPEN_API_KEY?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty string should error, yeah.

the OPENAI_API_KEY default is meant for when the field is omitted, not when someone explicitly blanks it. An explicit "" means they were mid-override and got it wrong, and falling back silently just turns that into a confusing auth failure further down. I'll split the two: omitted stays defaulted, empty or whitespace raises. what u think?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still wondering if user will declare on yaml api_key_env = " "
we will fall back to open api key default. but, he tried to override and maybe typo, so user experience in my view should error him.
Now we fallback

api_key = _resolve_openai_api_key(key_env_var)
if not api_key:
raise ModelConfigurationError(
"OpenAI provider requires OPENAI_API_KEY. "
f"OpenAI-compatible provider requires {key_env_var}. "
"Set it in your environment before running agentctl."
)

Expand All @@ -223,14 +250,17 @@ def _build_openai_model(
) from exc

# OpenAI's chat model uses the same `max_tokens` name as extra's config, so
# no remapping is needed. The API key is passed explicitly so OPENAI_API_KEY
# is honored regardless of how the installed SDK resolves credentials.
# no remapping is needed. The API key is passed explicitly so the resolved
# env var is honored regardless of how the installed SDK resolves
# credentials. `base_url` repoints the client at any OpenAI-compatible
# endpoint; when unset, ChatOpenAI's own default (api.openai.com) applies.
try:
return ChatOpenAI(
**_without_none(
{
"model": name,
"api_key": api_key,
"base_url": base_url,
"temperature": temperature,
"max_tokens": max_tokens,
"top_p": top_p,
Expand All @@ -239,8 +269,8 @@ def _build_openai_model(
)
except Exception as exc:
raise ModelConfigurationError(
"Could not initialize OpenAI chat model. Verify OPENAI_API_KEY is valid "
"and the model name is an OpenAI model your key can access."
f"Could not initialize OpenAI-compatible chat model. Verify {key_env_var} is "
"valid and the model name is one your key/endpoint can access."
) from exc


Expand All @@ -257,8 +287,8 @@ def _resolve_gemini_api_key() -> str | None:
return key.strip() if key and key.strip() else None


def _resolve_openai_api_key() -> str | None:
key = os.getenv("OPENAI_API_KEY")
def _resolve_openai_api_key(env_var: str = "OPENAI_API_KEY") -> str | None:
key = os.getenv(env_var)
return key.strip() if key and key.strip() else None


Expand Down
70 changes: 70 additions & 0 deletions src/agent_engine/models/presets.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
"""Known OpenAI-compatible vendor presets.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this file is needed. what you have built, its generic infra for custom models.
This led us to maintain code and each changes of the providers. let user declare api key name and urls and models, they write once time yaml and use model name each place

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the maintenance worry is fair but smaller than it looks I think

these base URLs don't move, and the table is six entries. The generic base_url/api_key_env path already covers everything the registry doesn't, including overriding a preset, so the registry adds no capability, it just saves every user re-typing the same two known values for a vendor the library could already name, or at least that was my initial tought. But if the team would rather ship zero vendor knowledge though, killing presets.py is one commit since the generic path stands on its own, lmk

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Asaf-prog wdyt? we need more opinion

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secondly i feel good with that. But we have to doc it also on official docs mdx files. focused and clear


Each preset supplies the ``base_url`` and ``api_key_env`` for a vendor whose
API speaks the OpenAI chat completions protocol, so ``provider: <preset id>``
in YAML resolves those two fields automatically instead of requiring them to
be typed out by hand every time. A YAML ``model`` block can still override
``base_url`` and/or ``api_key_env`` for a listed vendor (a self-hosted proxy
in front of it, a non-default key variable name, ...), and ``provider:
openai`` with explicit ``base_url``/``api_key_env`` remains the escape hatch
for any vendor or self-hosted server not listed here.

Presets deliberately do not carry a default model id: no provider in this
factory picks a model on the caller's behalf, and vendor model catalogs
change independently of this codebase, so ``name`` stays required in YAML
exactly as it is for every other provider.
"""

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class OpenAICompatPreset:
id: str
name: str
base_url: str
api_key_env: str


_PRESET_LIST: tuple[OpenAICompatPreset, ...] = (
OpenAICompatPreset(
id="zai",
name="Z.AI",
base_url="https://api.z.ai/api/coding/paas/v4",
api_key_env="ZAI_API_KEY",
),
OpenAICompatPreset(
id="deepseek",
name="DeepSeek",
base_url="https://api.deepseek.com/v1",
api_key_env="DEEPSEEK_API_KEY",
),
OpenAICompatPreset(
id="moonshot",
name="Moonshot",
base_url="https://api.moonshot.ai/v1",
api_key_env="MOONSHOT_API_KEY",
),
OpenAICompatPreset(
id="groq",
name="Groq",
base_url="https://api.groq.com/openai/v1",
api_key_env="GROQ_API_KEY",
),
OpenAICompatPreset(
id="xai",
name="xAI",
base_url="https://api.x.ai/v1",
api_key_env="XAI_API_KEY",
),
OpenAICompatPreset(
id="openrouter",
name="OpenRouter",
base_url="https://openrouter.ai/api/v1",
api_key_env="OPENROUTER_API_KEY",
),
)

OPENAI_COMPAT_PRESETS: dict[str, OpenAICompatPreset] = {p.id: p for p in _PRESET_LIST}
16 changes: 15 additions & 1 deletion src/agent_engine/parsers/yaml/parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
from agent_engine.runtime.hooks.models import HOOK_POINTS

_SECRET_MARKERS = ("api_key", "apikey", "secret", "token", "password", "private_key")
_SECRET_KEY_EXEMPTIONS = {"max_tokens"}
_SECRET_KEY_EXEMPTIONS = {"max_tokens", "api_key_env"}

# For *values*, a marker word alone is not evidence — ordinary prose like
# "Handles password reset requests" must pass. Flag a string value only when it
Expand All @@ -50,6 +50,7 @@
r"|-----BEGIN [A-Z ]*PRIVATE KEY-----" # PEM private keys
)
_SUPPORTED_MODEL_PROVIDERS = ("anthropic", "bedrock", "gemini", "openai")
_FIXED_ENDPOINT_PROVIDERS = ("anthropic", "bedrock", "gemini")


def _validate_plugins(plugins: Any, errors: list[ValidationError]) -> None:
Expand Down Expand Up @@ -349,6 +350,17 @@ def _validate_model(path: str, raw: Any, errors: list[ValidationError]) -> None:
)
)

if provider in _FIXED_ENDPOINT_PROVIDERS:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of nested ifs lets try make it better like (pseudo) :
for field_name in ("base_url", "api_key_env"):
if raw.get(field_name) is not None && provider in _FIXED_ENDPOINT_PROVIDERS
return error

for field_name in ("base_url", "api_key_env"):
if raw.get(field_name) is not None:
errors.append(
ValidationError(
f"{path}.{field_name}",
f"Not supported for provider '{provider}'; only OpenAI-compatible "
"providers accept a custom endpoint or key variable.",

@AmitAvital1 AmitAvital1 Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on error message: base url or api key naming variable?

)
)

name = raw.get("name")
if not isinstance(name, str) or not name.strip():
errors.append(ValidationError(f"{path}.name", "Required non-empty string"))
Expand Down Expand Up @@ -604,6 +616,8 @@ def _build_model(self, raw: dict[str, Any]) -> ModelConfig:
region=raw.get("region"),
max_tokens=raw.get("max_tokens"),
top_p=raw.get("top_p"),
base_url=raw.get("base_url"),
api_key_env=raw.get("api_key_env"),
)

def _build_resolvers(
Expand Down
Loading