diff --git a/articles/flow/ai-support/listeners.adoc b/articles/flow/ai-support/listeners.adoc index 08966b03e5..a0ceed8807 100644 --- a/articles/flow/ai-support/listeners.adoc +++ b/articles/flow/ai-support/listeners.adoc @@ -49,7 +49,7 @@ The response listener is called when the turn ends -- normally when the assistan The event carries: * [methodname]`getResponse()` -- the assistant's response text. On success, it may be empty when the model emitted only tool calls; on failure, it's always empty -- partial text received before the error isn't passed on, and the Message List shows a generic error message in its place. Empty responses are not appended to the conversation history. -* [methodname]`getError()` -- the failure cause, or an empty optional on success. +* [methodname]`getError()` -- the failure cause, or an empty optional on success. A turn stopped at a <> fails with a [classname]`ToolCallLimitExceededException`. * [methodname]`getMetadata()` -- the provider's metadata for the turn; see <<#response-metadata,Response Metadata>>. A typical use is persisting the conversation after each exchange -- see <>. diff --git a/articles/flow/ai-support/llm-providers.adoc b/articles/flow/ai-support/llm-providers.adoc index acacad07da..42af70d07e 100644 --- a/articles/flow/ai-support/llm-providers.adoc +++ b/articles/flow/ai-support/llm-providers.adoc @@ -104,6 +104,46 @@ The orchestrator sends its own system prompt on every turn, which replaces any [ [classname]`LangChain4JLLMProvider` is built from a [classname]`ChatModel` or [classname]`StreamingChatModel` and runs the tool-calling loop itself, so LangChain4j's [classname]`McpToolProvider` -- which attaches to an AI service built with [classname]`AiServices` -- has nowhere to plug in. With LangChain4j, MCP tools currently require a <<#custom-llm-providers,custom LLM provider>> built around an AI service, which then gets the tool provider, memory, and tool loop from LangChain4j. With Spring AI, the [classname]`ChatClient` setup above is all that's needed. +[[tool-call-limits]] +[role="since:com.vaadin:vaadin@V25.3"] +== Tool Call Limits + +A turn that uses <> is a loop: the model asks for tool calls, the provider runs them and calls the model again with the results, until the model answers. Both built-in providers bound that loop per turn. By default, the model may call any single tool at most 40 times and all tools together at most 150 times in one turn. Without a bound, a request the model can't satisfy keeps calling the model and the tools until the application is stopped. + +When a limit is exceeded, the turn fails with a [classname]`ToolCallLimitExceededException`, whichever provider runs it. The Message List shows its generic error message, and the exception reaches the <> and [methodname]`AIController.onResponse()` as the error of the turn. Its message names the limit that was exceeded and, for a per-tool limit, the tool: + +[source,java] +---- +.withResponseListener(event -> { + event.getError() + .filter(ToolCallLimitExceededException.class::isInstance) + .ifPresent(error -> log.warn( + "Turn stopped: {}", error.getMessage())); +}) +---- + +=== LangChain4j + +[classname]`LangChain4JLLMProvider` runs the loop itself and enforces the limits. Change them with [methodname]`setMaxCallsPerTool()` and [methodname]`setMaxTotalToolCalls()`. A value of `0` removes that limit. The values are read when a turn starts, so a change applies from the next prompt. + +[source,java] +---- +LangChain4JLLMProvider provider = new LangChain4JLLMProvider(chatModel); +provider.setMaxCallsPerTool(10); +provider.setMaxTotalToolCalls(0); // No limit across all tools +---- + +The provider checks the limits before running the tool calls of a response. When a call would take a count past its limit, none of the tool calls in that response run, and the model isn't called again. The failed turn leaves only its prompt in the provider's memory: tool calls and their results are sent to the model only within the turn they belong to and are never kept between turns, so the next prompt continues from where the last completed turn left the conversation. + +=== Spring AI + +[classname]`SpringAILLMProvider` adds no limit of its own. Spring AI bounds the loop with the same defaults, 40 calls per tool and 150 in total per turn, with either constructor of the provider. When Spring AI stops the loop, the provider fails the turn with the [classname]`ToolCallLimitExceededException` described above, carrying Spring AI's own message about the limit. That message may remain in the chat memory, with either constructor: the provider doesn't rewrite what Spring AI's advisors stored. + +The limits belong to the [classname]`ChatClient`, so tune or remove them there and pass the client to the [classname]`ChatClient` constructor of the provider; the [classname]`ChatModel` constructor builds a client with Spring AI's defaults. How to configure them is described under https://docs.spring.io/spring-ai/reference/api/tools.html#tool-call-limits[Tool Call Limits] in the Spring AI documentation. + +A <<#custom-llm-providers,custom provider>> decides itself whether, and how, it bounds its loop. + + [[background-execution]] [role="since:com.vaadin:vaadin@V25.3"] == Background Execution diff --git a/articles/flow/ai-support/tool-calling.adoc b/articles/flow/ai-support/tool-calling.adoc index c914643c8f..44c2b3ead8 100644 --- a/articles/flow/ai-support/tool-calling.adoc +++ b/articles/flow/ai-support/tool-calling.adoc @@ -58,6 +58,10 @@ For a reusable set of tools that does not depend on a specific LLM framework's a [NOTE] Tool objects registered via [methodname]`withTools()` are executed by the vendor framework, whose own error handling decides what the LLM sees when a tool throws -- by default, both LangChain4j and Spring AI relay the raw message of any exception. To control what the LLM learns about failures, define the tool through a controller instead and throw a [classname]`ToolException` for messages the LLM is meant to see; see <>. +.Tool Call Limits +[NOTE] +The tool-calling loop of a turn is bounded: by default, the model may call any single tool at most 40 times and all tools together at most 150 times in one turn. Exceeding a limit fails the turn with a [classname]`ToolCallLimitExceededException`, with either built-in provider. See <>. + == Programmatic Prompts