Using Agent Framework with llama-server locally with no server-side store, problems with tool calling

[mvx-team]

I have been trying to get the latest Agent Framework release running with llama-server from the llama.cpp project. I have tried local reasoning models like the Gemma 4 series and Qwen 3.8 27B.

Running an agent without any tool calls works fine using OpenAIChatClient. Adding tool calling into the agent as below, with store set to false results in an error. Note: store has to be False because llama-server does not implement any server-side message history.

agent_main = Agent(
client=local_chat_client,
instructions=write_prompt(),
name="LocalAgent",
tools=get_weather(),
default_options={"store": False}
)

Error message:
Agent framework failure: Stateless replay cannot reconstruct reasoning item(s) rs_xxxxxxxxxxxxxxx for call(s) call_xxxxxxxxxxxxxxx because encrypted reasoning content is missing. Use service-side continuation or explicitly configured atomic compaction to exclude each complete reasoning/tool-call group

Is there a way to pass the entire system prompt and tool call result back to llama-server? I have tried compaction using ToolResultCompactionStrategy(keep_last_tool_call_groups=0) but the tool call result is truncated.

---

I think the issue is related to #7577 and #7536 where the reasoning.encrypted_content field is automatically requested for reasoning models when using OpenAIChatClient.

llama-server doesn't support that field because the reasoning trace is in plaintext. See llama-server log below:

data: {"type":"response.completed","response":{"id":"resp_3WgCn4AtwZX96OdAzvS14G5Sq52j9qGx","object":"response","created_at":1792309608,"status":"completed","model":"C:\gemma-4-31B-it.gguf","output":[{"id":"rs_k8rAEQ0ViGLsndB9JEmAB9x9vzqaKHd8","summary":[],"type":"reasoning","content":[{"text":"The user wants to go to Seattle tomorrow morning. To provide a helpful travel plan, I should look for flights to Seattle and check the weather forecast for Seattle.\n\nFirst, I need to find flights to Seattle.\nSecond, I need to get the weather forecast for Seattle.","type":"reasoning_text"}],"encrypted_content":""},{"id":"fc_efLBKgsvERLnLOuqA9TvvhAc6cEVHwwD","type":"function_call","status":"completed","arguments":"{"location":"Seattle"}","call_id":"call_efLBKgsvERLnLOuqA9TvvhAc6cEVHwwD","name":"get_flights"},{"id":"fc_GbakRoTUI8G52ToPeSPU2JYIXX2r5YDj","type":"function_call","status":"completed","arguments":"{"location":"Seattle"}","call_id":"call_GbakRoTUI8G52ToPeSPU2JYIXX2r5YDj","name":"get_weather"}],...

[bomtobe-hgkim]

I checked the current source for ToolResultCompactionStrategy: keep_last_tool_call_groups=0 is documented as removing all included tool-call groups. That may explain the missing result; try retaining at least one group and verify the replay keeps the paired call and result. If the rs_ error persists, the framework error says stateless replay cannot reconstruct missing encrypted reasoning; resending the system prompt cannot recreate that item. Its stated alternatives are provider-side continuation or atomic compaction that excludes the complete reasoning/tool-call group. Source: https://github.com/microsoft/agent-framework/blob/main/python/packages/core/agent_framework/_compaction.py . I work with BOMTOBE on Orke and haven’t reproduced your llama-server setup; this is based on the source and error text.

[mvx-team]

Hi, thank you for the message. I forgot to add the compaction field in the agent example above but it's in my actual project. If I add ToolResultCompactionStrategy as below, the tool call result gets truncated.

agent_main = Agent( client=local_chat_client, instructions=write_prompt(), name="LocalAgent", tools=get_weather, compaction_strategy=ToolResultCompactionStrategy(keep_last_tool_call_groups=0) default_options={"store": False} )

If I use other compaction methods or no compaction method, I get this error:

Agent framework failure: Stateless replay cannot reconstruct reasoning item(s) rs_xxxxxxxxxxxxxxx for call(s) call_xxxxxxxxxxxxxxx because encrypted reasoning content is missing. Use service-side continuation or explicitly configured atomic compaction to exclude each complete reasoning/tool-call group

The issue at hand is that llama-server only exposes a basic OpenAI-compatible Responses endpoint.

[Eric Curtin (ericcurtin)]

With store=False, OpenAIChatClient (Responses API) asks for reasoning.encrypted_content, which llama-server returns empty, hence the replay error. The Chat Completions client has no such check:

from agent_framework.openai import OpenAIChatCompletionClient

local_chat_client = OpenAIChatCompletionClient(
    model="local", base_url="http://localhost:8080/v1", api_key="none"
)

You might also like llmman for running local models and agents (I maintain it).