diff --git a/docs/content/services/chat-service.md b/docs/content/services/chat-service.md index 46d2bad5..4596a9ba 100644 --- a/docs/content/services/chat-service.md +++ b/docs/content/services/chat-service.md @@ -81,7 +81,8 @@ ChatService chatService = ChatService.builder() | `timeout` | Duration | No | Request timeout (default: 60 seconds) | | `parameters` | ChatParameters | No | Default parameters applied to all requests | | `tools` | List\ | No | Default tools available to the model | -| `messageInterceptor` | MessageInterceptor\ | No | Modify assistant messages before returning | +| `messageInterceptor` | MessageInterceptor\ | No | Modify the complete assistant message before returning | +| `partialResponseInterceptor` | PartialResponseInterceptor\ | No | Modify each streamed content token before delivery | | `toolInterceptor` | ToolInterceptor\ | No | Normalize/modify tool call arguments | | `logRequests` | Boolean | No | Enable request logging (default: false) | | `logResponses` | Boolean | No | Enable response logging (default: false) | @@ -394,13 +395,13 @@ System.out.println(answer); ## Interceptors -Interceptors run automatically after every non-streaming response, before the result is returned to your application. They are configured once on the service builder and apply transparently to all subsequent calls. Both are `@FunctionalInterface`, so you can pass a lambda directly. +Interceptors are configured once on the service builder and apply transparently to all subsequent calls. Each one is a `@FunctionalInterface`, so you can pass a lambda directly. ### Message Interceptor `MessageInterceptor` lets you modify or sanitize the assistant's text content. Common uses: stripping whitespace, filtering unwanted patterns, normalizing formatting. -> **Note:** `MessageInterceptor` applies to **non-streaming** requests only. For streaming, process the content directly inside `ChatHandler` callbacks. +It is always called **once per assistant message, with the complete content**, in both modes. In **non-streaming** mode it runs before the response is returned to the caller. In **streaming** mode it runs on the aggregated message, before that message reaches `onCompleteResponse` and before the `CompletableFuture` returned by `chatStreaming` completes. Transformations that need the whole text, such as `strip()` or a pattern that spans several tokens, are therefore safe everywhere. ```java ChatService chatService = ChatService.builder() @@ -409,6 +410,21 @@ ChatService chatService = ChatService.builder() .build(); ``` +> **Note:** the tokens delivered to `onPartialResponse` are left untouched, so a caller that renders the stream sees the original text and only the final message carries the transformation. Use `PartialResponseInterceptor` when the tokens themselves have to change. + +### Partial Response Interceptor + +`PartialResponseInterceptor` intercepts each content token in **streaming** mode, before it is delivered to `onPartialResponse`. Common uses: masking or highlighting text as it appears, adapting tokens for a terminal or a UI widget. + +```java +ChatService chatService = ChatService.builder() + // ... + .partialResponseInterceptor((ctx, partialResponse) -> partialResponse.toUpperCase()) + .build(); +``` + +Every invocation receives one token exactly as the model streamed it. Tokens are never buffered, so a transformation that has to match across token boundaries belongs in a `MessageInterceptor` instead. This interceptor has no effect on non-streaming requests and does not alter the `ChatResponse` delivered to `onCompleteResponse`, which means the two hooks can be combined freely. + ### Tool Interceptor `ToolInterceptor` intercepts each completed tool call before it reaches your handler, letting you validate, normalize, or unwrap the arguments (for example, unwrapping double-encoded JSON strings that some models produce). @@ -428,15 +444,15 @@ ChatService chatService = ChatService.builder() ### InterceptorContext -Both interceptors receive an `InterceptorContext` as their first argument, which provides access to the current request, the current response, and a way to invoke the model again. +Every interceptor receives an `InterceptorContext` as its first argument, which provides access to the current request, the current response, and a way to invoke the model again. | Method | Description | |--------|-------------| | `ctx.request()` | The original `ChatRequest` that triggered this response | -| `ctx.response()` | An `Optional` with the current response | +| `ctx.response()` | An `Optional` with the current response, empty in `PartialResponseInterceptor` because no response exists yet | | `ctx.invoke(ChatRequest)` | Sends a new request to the model and returns its response | -`MessageInterceptor`, `ToolInterceptor`, and `InterceptorContext` are parameterized by the request type of the service they are registered on, so `ctx.request()` returns that concrete type with no cast: `ChatRequest` here, `DeploymentChatRequest` on [`DeploymentService`](/services/deployment-service), and `ModelGatewayChatRequest` on [`ModelGatewayChatService`](/services/model-gateway). The type argument is inferred when you pass a lambda, and only needs to be written out if you declare the interceptor separately: +`MessageInterceptor`, `PartialResponseInterceptor`, `ToolInterceptor`, and `InterceptorContext` are parameterized by the request type of the service they are registered on, so `ctx.request()` returns that concrete type with no cast: `ChatRequest` here, `DeploymentChatRequest` on [`DeploymentService`](/services/deployment-service), and `ModelGatewayChatRequest` on [`ModelGatewayChatService`](/services/model-gateway). The type argument is inferred when you pass a lambda, and only needs to be written out if you declare the interceptor separately: ```java MessageInterceptor interceptor = (ctx, message) -> message == null ? "" : message.strip(); diff --git a/docs/content/services/deployment-service.md b/docs/content/services/deployment-service.md index 070d55c3..e127c8ce 100644 --- a/docs/content/services/deployment-service.md +++ b/docs/content/services/deployment-service.md @@ -69,7 +69,8 @@ All routing is done through the `deploymentId` in each request, so no `projectId | `verifySsl` | Boolean | No | SSL certificate verification (default: true) | | `parameters` | ChatParameters | No | Default chat parameters applied to every chat request | | `tools` | List\ | No | Default tools available to the model | -| `messageInterceptor` | MessageInterceptor\ | No | Post-processing hook for the assistant's text content | +| `messageInterceptor` | MessageInterceptor\ | No | Post-processing hook for the complete assistant message | +| `partialResponseInterceptor` | PartialResponseInterceptor\ | No | Post-processing hook for each streamed content token | | `toolInterceptor` | ToolInterceptor\ | No | Post-processing hook for function call arguments | > Either `apiKey` or `authenticator` must be provided. `projectId`, `spaceId`, and `modelId` are **ignored**, and a warning is logged if they are set on a request's parameters object. diff --git a/docs/content/services/model-gateway/chat.md b/docs/content/services/model-gateway/chat.md index dc61937b..b0109109 100644 --- a/docs/content/services/model-gateway/chat.md +++ b/docs/content/services/model-gateway/chat.md @@ -31,7 +31,7 @@ System.out.println(response.toAssistantMessage().content()); - Send synchronous and streaming chat requests to any model available through the gateway. - Use gateway-specific parameters such as service tier, reasoning effort, audio modalities, caching, and routing configuration. -- Apply `MessageInterceptor` and `ToolInterceptor` for post-processing. +- Apply `MessageInterceptor`, `PartialResponseInterceptor`, and `ToolInterceptor` for post-processing. - Read gateway metadata on every response: `serviceTier()`, `systemFingerprint()`, and `cached()`. --- @@ -58,7 +58,8 @@ ModelGatewayChatService service = ModelGatewayChatService.builder() | `modelId` | String | Yes | Third-party model identifier (e.g., `"gpt-4o"`, `"claude-3-5-sonnet"`) | | `parameters` | ModelGatewayChatParameters | No | Default parameters applied to every request | | `tools` | List\ | No | Default tools available to the model | -| `messageInterceptor` | MessageInterceptor\ | No | Post-processing hook for the assistant's text content | +| `messageInterceptor` | MessageInterceptor\ | No | Post-processing hook for the complete assistant message | +| `partialResponseInterceptor` | PartialResponseInterceptor\ | No | Post-processing hook for each streamed content token | | `toolInterceptor` | ToolInterceptor\ | No | Post-processing hook for function call arguments | | `timeout` | Duration | No | Default request timeout (default: 60 seconds) | | `logRequests` | Boolean | No | Enable request logging (default: false) | @@ -341,7 +342,7 @@ System.out.println("Total tokens: " + response.usage().totalTokens()); ## Interceptors -Interceptors work identically to how they work in `ChatService`. See the [Chat Service - Interceptors](../../services/chat-service#interceptors) section for the full description of `MessageInterceptor`, `ToolInterceptor`, and `InterceptorContext`. +Interceptors work identically to how they work in `ChatService`. See the [Chat Service - Interceptors](../../services/chat-service#interceptors) section for the full description of `MessageInterceptor`, `PartialResponseInterceptor`, `ToolInterceptor`, and `InterceptorContext`. ```java ModelGatewayChatService service = ModelGatewayChatService.builder() diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/ChatClientContext.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/ChatClientContext.java index 64d72cf3..ec3cc859 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/ChatClientContext.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/ChatClientContext.java @@ -4,6 +4,8 @@ */ package com.ibm.watsonx.ai.chat; +import com.ibm.watsonx.ai.chat.interceptor.MessageInterceptor; +import com.ibm.watsonx.ai.chat.interceptor.PartialResponseInterceptor; import com.ibm.watsonx.ai.chat.interceptor.ToolInterceptor; import com.ibm.watsonx.ai.chat.model.ExtractionTags; @@ -15,12 +17,16 @@ public class ChatClientContext { private final ChatProvider chatProvider; private final R chatRequest; + private final MessageInterceptor messageInterceptor; + private final PartialResponseInterceptor partialResponseInterceptor; private final ToolInterceptor toolInterceptor; private final ExtractionTags extractionTags; private ChatClientContext(Builder builder) { chatProvider = builder.chatProvider; chatRequest = builder.chatRequest; + messageInterceptor = builder.messageInterceptor; + partialResponseInterceptor = builder.partialResponseInterceptor; toolInterceptor = builder.toolInterceptor; extractionTags = builder.extractionTags; } @@ -43,6 +49,24 @@ public R chatRequest() { return chatRequest; } + /** + * Returns the message interceptor. + * + * @return the message interceptor + */ + public MessageInterceptor messageInterceptor() { + return messageInterceptor; + } + + /** + * Returns the partial response interceptor. + * + * @return the partial response interceptor + */ + public PartialResponseInterceptor partialResponseInterceptor() { + return partialResponseInterceptor; + } + /** * Returns the tool interceptor. * @@ -79,6 +103,8 @@ public static Builder builder() { public static class Builder { private ChatProvider chatProvider; private R chatRequest; + private MessageInterceptor messageInterceptor; + private PartialResponseInterceptor partialResponseInterceptor; private ToolInterceptor toolInterceptor; private ExtractionTags extractionTags; @@ -104,6 +130,26 @@ public Builder chatRequest(R chatRequest) { return this; } + /** + * Sets the message interceptor. + * + * @param messageInterceptor the message interceptor + */ + public Builder messageInterceptor(MessageInterceptor messageInterceptor) { + this.messageInterceptor = messageInterceptor; + return this; + } + + /** + * Sets the partial response interceptor. + * + * @param partialResponseInterceptor the partial response interceptor + */ + public Builder partialResponseInterceptor(PartialResponseInterceptor partialResponseInterceptor) { + this.partialResponseInterceptor = partialResponseInterceptor; + return this; + } + /** * Sets the tool interceptor. * diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/ChatService.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/ChatService.java index baf7ee68..e2f8a9bb 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/ChatService.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/ChatService.java @@ -15,6 +15,7 @@ import com.ibm.watsonx.ai.WatsonxService.CryptoService; import com.ibm.watsonx.ai.chat.interceptor.InterceptorContext; import com.ibm.watsonx.ai.chat.interceptor.MessageInterceptor; +import com.ibm.watsonx.ai.chat.interceptor.PartialResponseInterceptor; import com.ibm.watsonx.ai.chat.interceptor.ToolInterceptor; import com.ibm.watsonx.ai.chat.model.BaseChatParameters.ToolChoiceOption; import com.ibm.watsonx.ai.chat.model.ChatMessage; @@ -51,6 +52,7 @@ public class ChatService extends CryptoService implements ChatProvider { private final ChatRestClient client; private final MessageInterceptor messageInterceptor; + private final PartialResponseInterceptor partialResponseInterceptor; private final ToolInterceptor toolInterceptor; private final ChatProvider chatProvider; private final ChatParameters defaultParameters; @@ -60,6 +62,7 @@ private ChatService(Builder builder) { super(builder); requireNonNull(builder.authenticator(), "authenticator cannot be null"); messageInterceptor = builder.messageInterceptor; + partialResponseInterceptor = builder.partialResponseInterceptor; toolInterceptor = builder.toolInterceptor; defaultTools = isNull(builder.defaultTools) ? null : List.copyOf(builder.defaultTools); @@ -85,7 +88,7 @@ private ChatService(Builder builder) { .authenticator(builder.authenticator()) .build(); - chatProvider = nonNull(messageInterceptor) || nonNull(toolInterceptor) + chatProvider = nonNull(messageInterceptor) || nonNull(partialResponseInterceptor) || nonNull(toolInterceptor) ? builder.copyWithoutInterceptors().parameters(defaultParameters).build() : null; } @@ -153,6 +156,8 @@ public CompletableFuture chatStreaming(ChatRequest chatRequest, Ch var context = ChatClientContext.builder() .chatProvider(chatProvider) .chatRequest(chatRequest) + .messageInterceptor(messageInterceptor) + .partialResponseInterceptor(partialResponseInterceptor) .toolInterceptor(toolInterceptor) .extractionTags(extractionTags) .build(); @@ -391,6 +396,7 @@ public static Builder builder() { */ public final static class Builder extends CryptoService.Builder { private MessageInterceptor messageInterceptor; + private PartialResponseInterceptor partialResponseInterceptor; private ToolInterceptor toolInterceptor; private ChatParameters defaultParameters; private List defaultTools; @@ -413,14 +419,13 @@ public Builder parameters(ChatParameters parameters) { /** * Registers a {@link MessageInterceptor} used to modify or sanitize the assistant's textual content before it is returned to the caller. *

- * This interceptor is invoked on the final aggregated content (non-streaming responses only), allowing adjustments such as rewriting, - * filtering, or normalization. + * The interceptor is invoked once per assistant message with the complete content, both on non-streaming responses and on the aggregated + * message of a streaming session. *

* Example: * *

{@code
-         * ChatService.builder()
-         *     .messageInterceptor((request, content) -> content.replace("error", "issue"));
+         * ChatService.builder().messageInterceptor((ctx, content) -> content.replace("error", "issue"));
          * }
* * @param messageInterceptor the interceptor to apply @@ -430,6 +435,26 @@ public Builder messageInterceptor(MessageInterceptor messageInterce return this; } + /** + * Registers a {@link PartialResponseInterceptor} used to modify each partial response before it is delivered to + * {@link ChatHandler#onPartialResponse}. + *

+ * This interceptor has no effect on non-streaming requests and does not alter the {@link ChatResponse} delivered to + * {@link ChatHandler#onCompleteResponse}. + *

+ * Example: + * + *

{@code
+         * ChatService.builder().partialResponseInterceptor((ctx, partialResponse) -> partialResponse.toUpperCase());
+         * }
+ * + * @param partialResponseInterceptor the interceptor to apply + */ + public Builder partialResponseInterceptor(PartialResponseInterceptor partialResponseInterceptor) { + this.partialResponseInterceptor = partialResponseInterceptor; + return this; + } + /** * Registers a {@link ToolInterceptor} to modify or normalize function call arguments before tool execution. *

diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/DefaultRestClient.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/DefaultRestClient.java index d32e0105..638c58c6 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/DefaultRestClient.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/DefaultRestClient.java @@ -84,7 +84,8 @@ public CompletableFuture chatStreaming( var chatSubscriber = new DefaultChatSubscriber( new SseEventProcessor(textChatRequest.tools(), context.extractionTags(), TextChatResponse::builder), - new ChatHandlerDecorator<>(handler, interceptorContext, context.toolInterceptor()) + new ChatHandlerDecorator<>(handler, interceptorContext, context.messageInterceptor(), context.partialResponseInterceptor(), + context.toolInterceptor()) ); var subscriber = chatSubscriber.asFlowSubscriber(response, !handler.failOnFirstError()); diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/decorator/ChatHandlerDecorator.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/decorator/ChatHandlerDecorator.java index 4ba92ee9..ae5689db 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/decorator/ChatHandlerDecorator.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/decorator/ChatHandlerDecorator.java @@ -14,6 +14,8 @@ import com.ibm.watsonx.ai.chat.ChatHandler; import com.ibm.watsonx.ai.chat.ChatResponse; import com.ibm.watsonx.ai.chat.interceptor.InterceptorContext; +import com.ibm.watsonx.ai.chat.interceptor.MessageInterceptor; +import com.ibm.watsonx.ai.chat.interceptor.PartialResponseInterceptor; import com.ibm.watsonx.ai.chat.interceptor.ToolInterceptor; import com.ibm.watsonx.ai.chat.model.CompletedToolCall; import com.ibm.watsonx.ai.chat.model.PartialChatResponse; @@ -29,7 +31,7 @@ *

    *
  • Sequential delivery of every callback (partial responses, partial thinking, partial and complete tool calls, complete responses, errors), one * at a time and in the order they were emitted
  • - *
  • Optional interception of complete tool calls, applied before they are delivered
  • + *
  • Optional interception of partial responses, complete tool calls and the complete message, applied before they are delivered
  • *
  • Callback scheduling using {@link CompletableFuture}, so the emitting thread is never blocked by user code
  • *
* @@ -54,6 +56,16 @@ public class ChatHandlerDecorator implements ChatHand */ private final InterceptorContext context; + /** + * Optional interceptor for modifying the complete message before it reaches the delegate. + */ + private final MessageInterceptor messageInterceptor; + + /** + * Optional interceptor for modifying each partial response before it reaches the delegate. + */ + private final PartialResponseInterceptor partialResponseInterceptor; + /** * Optional interceptor for modifying or validating tool calls before they reach the delegate. */ @@ -80,17 +92,22 @@ public class ChatHandlerDecorator implements ChatHand * * @param delegate the underlying chat handler to receive decorated callbacks * @param context the interceptor context for tool call processing + * @param messageInterceptor optional interceptor for the complete message + * @param partialResponseInterceptor optional interceptor for partial responses * @param toolInterceptor optional interceptor for tool calls */ - public ChatHandlerDecorator(ChatHandler delegate, InterceptorContext context, ToolInterceptor toolInterceptor) { + public ChatHandlerDecorator(ChatHandler delegate, InterceptorContext context, MessageInterceptor messageInterceptor, + PartialResponseInterceptor partialResponseInterceptor, ToolInterceptor toolInterceptor) { this.delegate = delegate; this.context = context; + this.messageInterceptor = messageInterceptor; + this.partialResponseInterceptor = partialResponseInterceptor; this.toolInterceptor = toolInterceptor; } @Override public void onPartialResponse(String partialResponse, PartialChatResponse partialChatResponse) { - scheduleCallback(() -> delegate.onPartialResponse(partialResponse, partialChatResponse)); + scheduleCallback(() -> delegate.onPartialResponse(normalize(partialResponse), partialChatResponse)); } @Override @@ -159,6 +176,41 @@ public CompletableFuture> awaitCallbacks() { return callbackChain.get().thenApply(v -> List.copyOf(deliveredToolCalls)); } + /** + * Applies {@link #messageInterceptor} to the given response, falling back to the un-normalized response when the interceptor fails. + * + * @param response the response to normalize + * @return the response with the interceptor applied + */ + public ChatResponse normalize(ChatResponse response) { + if (isNull(messageInterceptor)) + return response; + + try { + return response.toBuilder() + .choices(messageInterceptor.intercept(context.withResponse(response))) + .build(); + } catch (RuntimeException | Error e) { + safeOnError(e); + return response; + } + } + + /** + * Applies {@link #partialResponseInterceptor} to the given token, falling back to the un-normalized token when the interceptor fails. + */ + private String normalize(String partialResponse) { + if (isNull(partialResponseInterceptor)) + return partialResponse; + + try { + return partialResponseInterceptor.intercept(context, partialResponse); + } catch (RuntimeException | Error e) { + safeOnError(e); + return partialResponse; + } + } + /** * Applies {@link #toolInterceptor} to the given tool call, falling back to the un-normalized tool call when the interceptor fails. */ diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/interceptor/InterceptorContext.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/interceptor/InterceptorContext.java index 76225207..56786e7f 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/interceptor/InterceptorContext.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/interceptor/InterceptorContext.java @@ -43,6 +43,16 @@ public Optional response() { return Optional.ofNullable(response); } + /** + * Returns a copy of this context bound to the given response. + * + * @param response the chat response to bind + * @return a new {@link InterceptorContext} carrying the given response + */ + public InterceptorContext withResponse(ChatResponse response) { + return new InterceptorContext<>(chatProvider, request, response); + } + /** * Sends a request to an LLM for auxiliary processing. *

diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/interceptor/PartialResponseInterceptor.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/interceptor/PartialResponseInterceptor.java new file mode 100644 index 00000000..606bb8b7 --- /dev/null +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/interceptor/PartialResponseInterceptor.java @@ -0,0 +1,37 @@ +/* + * Copyright 2025 IBM Corporation + * SPDX-License-Identifier: Apache-2.0 + */ +package com.ibm.watsonx.ai.chat.interceptor; + +import com.ibm.watsonx.ai.chat.BaseChatRequest; +import com.ibm.watsonx.ai.chat.ChatHandler; + +/** + * Functional interface for intercepting and modifying each partial response emitted during a streaming session. + *

+ * Every invocation receives a single content token exactly as the model streamed it, and the returned value is what + * {@link ChatHandler#onPartialResponse} delivers. Tokens are never buffered, so a transformation that has to match across token boundaries cannot be + * expressed here. Use {@link MessageInterceptor}, which is always applied to the complete message. + *

+ * Example usage: + * + *

{@code
+ * PartialResponseInterceptor interceptor =
+ *     (ctx, partialResponse) -> partialResponse.replace("foo", "bar");
+ * }
+ * + * @param the concrete chat request type handled by the intercepted provider + */ +@FunctionalInterface +public interface PartialResponseInterceptor { + + /** + * Intercepts and modifies a single partial response. + * + * @param ctx the interceptor context, providing access to the request and other contextual information + * @param partialResponse the content token to intercept + * @return the modified token + */ + String intercept(InterceptorContext ctx, String partialResponse); +} diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/streaming/DefaultChatSubscriber.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/streaming/DefaultChatSubscriber.java index c576b9c1..c88d71bf 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/streaming/DefaultChatSubscriber.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/chat/streaming/DefaultChatSubscriber.java @@ -83,11 +83,13 @@ public CompletableFuture onComplete() { return CompletableFuture.completedFuture(null); return awaitCallbacks() - .thenCompose(completeToolCalls -> { - var response = processor.buildResponse(); + .thenComposeAsync(completeToolCalls -> { + var builtResponse = processor.buildResponse(); - if (response.isBlockedByModeration()) - return CompletableFuture.failedFuture(new ModerationException(response.moderations())); + if (builtResponse.isBlockedByModeration()) + return CompletableFuture.failedFuture(new ModerationException(builtResponse.moderations())); + + ChatResponse response = decorator.normalize(builtResponse); if (nonNull(completeToolCalls) && !completeToolCalls.isEmpty()) { var choices = response.choices().stream() @@ -103,7 +105,7 @@ public CompletableFuture onComplete() { handler.onCompleteResponse(response); } return awaitCallbacks().thenApply(ignored -> response); - }); + }, ExecutorProvider.callbackExecutor()); } @Override diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/deployment/DefaultRestClient.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/deployment/DefaultRestClient.java index 44dc4cf8..bf5bdbd1 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/deployment/DefaultRestClient.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/deployment/DefaultRestClient.java @@ -214,7 +214,8 @@ public CompletableFuture chatStreaming( var chatSubscriber = new DefaultChatSubscriber( new SseEventProcessor(textChatRequest.tools(), context.extractionTags(), TextChatResponse::builder), - new ChatHandlerDecorator<>(handler, interceptorContext, context.toolInterceptor()) + new ChatHandlerDecorator<>(handler, interceptorContext, context.messageInterceptor(), context.partialResponseInterceptor(), + context.toolInterceptor()) ); var subscriber = chatSubscriber.asFlowSubscriber(response, !handler.failOnFirstError()); diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/deployment/DeploymentService.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/deployment/DeploymentService.java index 55b54b18..51b97768 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/deployment/DeploymentService.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/deployment/DeploymentService.java @@ -27,6 +27,7 @@ import com.ibm.watsonx.ai.chat.TextChatResponse; import com.ibm.watsonx.ai.chat.interceptor.InterceptorContext; import com.ibm.watsonx.ai.chat.interceptor.MessageInterceptor; +import com.ibm.watsonx.ai.chat.interceptor.PartialResponseInterceptor; import com.ibm.watsonx.ai.chat.interceptor.ToolInterceptor; import com.ibm.watsonx.ai.chat.model.BaseChatParameters.ToolChoiceOption; import com.ibm.watsonx.ai.chat.model.ChatMessage; @@ -71,6 +72,7 @@ public class DeploymentService extends WatsonxService private static final Logger logger = LoggerFactory.getLogger(DeploymentService.class); private final DeploymentRestClient client; private final MessageInterceptor messageInterceptor; + private final PartialResponseInterceptor partialResponseInterceptor; private final ToolInterceptor toolInterceptor; private final ChatProvider chatProvider; private final ChatParameters defaultParameters; @@ -80,6 +82,7 @@ private DeploymentService(Builder builder) { super(builder); requireNonNull(builder.authenticator(), "authenticator cannot be null"); messageInterceptor = builder.messageInterceptor; + partialResponseInterceptor = builder.partialResponseInterceptor; toolInterceptor = builder.toolInterceptor; defaultTools = isNull(builder.defaultTools) ? null : List.copyOf(builder.defaultTools); defaultParameters = requireNonNullElse(builder.defaultParameters, ChatParameters.builder().build()); @@ -95,7 +98,7 @@ private DeploymentService(Builder builder) { .verifySsl(verifySsl) .build(); - chatProvider = nonNull(messageInterceptor) || nonNull(toolInterceptor) + chatProvider = nonNull(messageInterceptor) || nonNull(partialResponseInterceptor) || nonNull(toolInterceptor) ? builder.copyWithoutInterceptors().parameters(defaultParameters).build() : null; } @@ -241,6 +244,8 @@ public CompletableFuture chatStreaming(DeploymentChatRequest chatR var context = ChatClientContext.builder() .chatProvider(chatProvider) .chatRequest(chatRequest) + .messageInterceptor(messageInterceptor) + .partialResponseInterceptor(partialResponseInterceptor) .toolInterceptor(toolInterceptor) .extractionTags(extractionTags) .build(); @@ -607,6 +612,7 @@ private void logIgnoredParameters(String modelId, String projectId, String space */ public final static class Builder extends WatsonxService.Builder { private MessageInterceptor messageInterceptor; + private PartialResponseInterceptor partialResponseInterceptor; private ToolInterceptor toolInterceptor; private ChatParameters defaultParameters; private List defaultTools; @@ -629,14 +635,13 @@ public Builder parameters(ChatParameters parameters) { /** * Registers a {@link MessageInterceptor} used to modify or sanitize the assistant's textual content before it is returned to the caller. *

- * This interceptor is invoked on the final aggregated content (non-streaming responses only), allowing adjustments such as rewriting, - * filtering, or normalization. + * The interceptor is invoked once per assistant message with the complete content, both on non-streaming responses and on the aggregated + * message of a streaming session. *

* Example: * *

{@code
-         * ChatService.builder()
-         *     .messageInterceptor((request, content) -> content.replace("error", "issue"));
+         * DeploymentService.builder().messageInterceptor((ctx, content) -> content.replace("error", "issue"));
          * }
* * @param messageInterceptor the interceptor to apply @@ -646,6 +651,26 @@ public Builder messageInterceptor(MessageInterceptor mess return this; } + /** + * Registers a {@link PartialResponseInterceptor} used to modify each partial response before it is delivered to + * {@link ChatHandler#onPartialResponse}. + *

+ * This interceptor has no effect on non-streaming requests and does not alter the {@link ChatResponse} delivered to + * {@link ChatHandler#onCompleteResponse}. + *

+ * Example: + * + *

{@code
+         * DeploymentService.builder().partialResponseInterceptor((ctx, partialResponse) -> partialResponse.toUpperCase());
+         * }
+ * + * @param partialResponseInterceptor the interceptor to apply + */ + public Builder partialResponseInterceptor(PartialResponseInterceptor partialResponseInterceptor) { + this.partialResponseInterceptor = partialResponseInterceptor; + return this; + } + /** * Registers a {@link ToolInterceptor} to modify or normalize function call arguments before tool execution. *

diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/gateway/chat/DefaultRestClient.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/gateway/chat/DefaultRestClient.java index e8745b90..d54aa119 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/gateway/chat/DefaultRestClient.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/gateway/chat/DefaultRestClient.java @@ -91,7 +91,8 @@ public CompletableFuture chatStreaming( var chatSubscriber = new DefaultChatSubscriber( new SseEventProcessor(gatewayRequest.tools(), context.extractionTags(), ModelGatewayChatResponse::builder), - new ChatHandlerDecorator<>(handler, interceptorContext, context.toolInterceptor()) + new ChatHandlerDecorator<>(handler, interceptorContext, context.messageInterceptor(), context.partialResponseInterceptor(), + context.toolInterceptor()) ); var subscriber = chatSubscriber.asFlowSubscriber(response, !handler.failOnFirstError()); diff --git a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/gateway/chat/ModelGatewayChatService.java b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/gateway/chat/ModelGatewayChatService.java index 4603b62e..ca6ffae7 100644 --- a/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/gateway/chat/ModelGatewayChatService.java +++ b/modules/watsonx-ai/src/main/java/com/ibm/watsonx/ai/gateway/chat/ModelGatewayChatService.java @@ -20,6 +20,7 @@ import com.ibm.watsonx.ai.chat.ExecutableTool; import com.ibm.watsonx.ai.chat.interceptor.InterceptorContext; import com.ibm.watsonx.ai.chat.interceptor.MessageInterceptor; +import com.ibm.watsonx.ai.chat.interceptor.PartialResponseInterceptor; import com.ibm.watsonx.ai.chat.interceptor.ToolInterceptor; import com.ibm.watsonx.ai.chat.model.ChatMessage; import com.ibm.watsonx.ai.chat.model.PartialChatResponse; @@ -55,6 +56,7 @@ public class ModelGatewayChatService extends WatsonxService implements ChatProvider { private final ModelGatewayChatRestClient client; private final MessageInterceptor messageInterceptor; + private final PartialResponseInterceptor partialResponseInterceptor; private final ToolInterceptor toolInterceptor; private final ChatProvider chatProvider; private final ModelGatewayChatParameters defaultParameters; @@ -66,6 +68,7 @@ private ModelGatewayChatService(Builder builder) { requireNonNull(builder.authenticator(), "authenticator cannot be null"); modelId = requireNonNull(builder.modelId, "The modelId must be provided"); messageInterceptor = builder.messageInterceptor; + partialResponseInterceptor = builder.partialResponseInterceptor; toolInterceptor = builder.toolInterceptor; defaultTools = isNull(builder.defaultTools) ? null : List.copyOf(builder.defaultTools); defaultParameters = builder.defaultParameters; @@ -81,7 +84,7 @@ private ModelGatewayChatService(Builder builder) { .verifySsl(verifySsl) .build(); - chatProvider = nonNull(messageInterceptor) || nonNull(toolInterceptor) + chatProvider = nonNull(messageInterceptor) || nonNull(partialResponseInterceptor) || nonNull(toolInterceptor) ? builder.copyWithoutInterceptors().parameters(defaultParameters).build() : null; } @@ -137,6 +140,8 @@ public CompletableFuture chatStreaming(ModelGatewayChatRequest cha var context = ChatClientContext.builder() .chatProvider(chatProvider) .chatRequest(chatRequest) + .messageInterceptor(messageInterceptor) + .partialResponseInterceptor(partialResponseInterceptor) .toolInterceptor(toolInterceptor) .build(); @@ -385,6 +390,7 @@ public static Builder builder() { public static final class Builder extends WatsonxService.Builder { private String modelId; private MessageInterceptor messageInterceptor; + private PartialResponseInterceptor partialResponseInterceptor; private ToolInterceptor toolInterceptor; private ModelGatewayChatParameters defaultParameters; private List defaultTools; @@ -416,6 +422,16 @@ public Builder parameters(ModelGatewayChatParameters parameters) { /** * Registers a {@link MessageInterceptor} used to modify or sanitize the assistant's textual content before it is returned to the caller. + *

+ * The interceptor is invoked once per assistant message with the complete content, both on non-streaming responses and on the aggregated + * message of a streaming session. + *

+ * Example: + * + *

{@code
+         * ModelGatewayChatService.builder()
+         *     .messageInterceptor((ctx, content) -> content.replace("error", "issue"));
+         * }
* * @param messageInterceptor the interceptor to apply */ @@ -424,6 +440,27 @@ public Builder messageInterceptor(MessageInterceptor me return this; } + /** + * Registers a {@link PartialResponseInterceptor} used to modify each partial response before it is delivered to + * {@link ChatHandler#onPartialResponse}. + *

+ * This interceptor has no effect on non-streaming requests and does not alter the {@link ChatResponse} delivered to + * {@link ChatHandler#onCompleteResponse}. + *

+ * Example: + * + *

{@code
+         * ModelGatewayChatService.builder()
+         *     .partialResponseInterceptor((ctx, partialResponse) -> partialResponse.toUpperCase());
+         * }
+ * + * @param partialResponseInterceptor the interceptor to apply + */ + public Builder partialResponseInterceptor(PartialResponseInterceptor partialResponseInterceptor) { + this.partialResponseInterceptor = partialResponseInterceptor; + return this; + } + /** * Registers a {@link ToolInterceptor} to modify or normalize function call arguments before tool execution. * diff --git a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/DeploymentServiceTest.java b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/DeploymentServiceTest.java index 7a74e82c..85895cad 100644 --- a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/DeploymentServiceTest.java +++ b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/DeploymentServiceTest.java @@ -38,6 +38,7 @@ import java.util.concurrent.CompletableFuture; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; import java.util.function.Consumer; import java.util.stream.Collectors; import org.junit.jupiter.api.BeforeEach; @@ -1506,7 +1507,7 @@ void should_override_assistant_content() { } @Test - void should_not_override_assistant_content_in_streaming() { + void should_apply_message_interceptor_to_the_complete_streaming_message() { String REQUEST = """ { @@ -1649,7 +1650,7 @@ public void onError(Throwable error) { var chatResponse = assertDoesNotThrow(() -> result.get(3, TimeUnit.SECONDS)); var assistantMessage = chatResponse.toAssistantMessage(); - assertEquals("Hello! I'm doing well, thank you. How can I assist you today?", assistantMessage.content()); + assertEquals("I don't feel good.", assistantMessage.content()); assertFalse(assistantMessage.hasToolCalls()); assertEquals("Hello! I'm doing well, thank you. How can I assist you today?", partialResponses.stream().collect(Collectors.joining())); } @@ -2461,6 +2462,83 @@ public void onPartialResponse(String partialResponse, PartialChatResponse partia assertEquals(2, requestBodies.stream().filter(body -> ((List) body.get("messages")).size() == 1).count()); } + @Test + void should_apply_message_and_partial_response_interceptors_in_streaming() { + + wireMock.stubFor(post("/ml/v1/deployments/my-deployment-id/text/chat_stream?version=%s".formatted(API_VERSION)) + .willReturn(aResponse() + .withStatus(200) + .withChunkedDribbleDelay(5, 200) + .withBody( + """ + id: 1 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant","content":"Hello"}}],"created":1764692529} + + id: 2 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[{"index":0,"finish_reason":null,"delta":{"content":" wor"}}],"created":1764692529} + + id: 3 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[{"index":0,"finish_reason":null,"delta":{"content":"ld!"}}],"created":1764692529} + + id: 4 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[{"index":0,"finish_reason":"stop","delta":{"content":""}}],"created":1764692529} + + id: 5 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[],"created":1764692529,"usage":{"completion_tokens":3,"prompt_tokens":10,"total_tokens":13}} + """))); + + when(mockAuthenticator.tokenAsync()).thenReturn(completedFuture("my-super-token")); + + var intercepted = new AtomicReference(); + var deploymentService = DeploymentService.builder() + .baseUrl(URI.create("http://localhost:%s".formatted(wireMock.getPort()))) + .authenticator(mockAuthenticator) + .messageInterceptor((ctx, message) -> { + intercepted.set(message); + return message.replace("Hello world", "Ciao mondo"); + }) + .partialResponseInterceptor((ctx, partialResponse) -> partialResponse.toUpperCase()) + .build(); + + var request = DeploymentChatRequest.builder() + .messages(UserMessage.text("Hi")) + .deploymentId("my-deployment-id") + .build(); + + List partialResponses = new ArrayList<>(); + CompletableFuture delivered = new CompletableFuture<>(); + var future = deploymentService.chatStreaming(request, new ChatHandler() { + + @Override + public void onPartialResponse(String partialResponse, PartialChatResponse partialChatResponse) { + partialResponses.add(partialResponse); + } + + @Override + public void onCompleteResponse(ChatResponse completeResponse) { + delivered.complete(completeResponse); + } + + @Override + public void onError(Throwable error) { + delivered.completeExceptionally(error); + } + }); + + var returnedResponse = assertDoesNotThrow(() -> future.get(3, TimeUnit.SECONDS)); + var completeResponse = assertDoesNotThrow(() -> delivered.get(3, TimeUnit.SECONDS)); + + assertEquals("Hello world!", intercepted.get()); + assertEquals(List.of("HELLO", " WOR", "LD!"), partialResponses); + assertEquals("Ciao mondo!", completeResponse.toAssistantMessage().content()); + assertEquals("Ciao mondo!", returnedResponse.toAssistantMessage().content()); + } + /** * Awaits the streaming response and asserts that both the aggregated response and the partial chunks contain the expected content. */ diff --git a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/ChatServiceStreamingTest.java b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/ChatServiceStreamingTest.java index ef741c83..2d22a704 100644 --- a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/ChatServiceStreamingTest.java +++ b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/ChatServiceStreamingTest.java @@ -106,6 +106,31 @@ @Isolated("Verifies executor/thread behavior with tight timeouts; must run without concurrent CPU contention.") public class ChatServiceStreamingTest extends AbstractWatsonxTest { + /** + * A minimal stream whose assistant message is split so that the words "Hello world" span three tokens. + */ + private static final String SPLIT_WORD_STREAMING_RESPONSE = + """ + id: 1 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant","content":"Hello"}}],"created":1764692529} + + id: 2 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[{"index":0,"finish_reason":null,"delta":{"content":" wor"}}],"created":1764692529} + + id: 3 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[{"index":0,"finish_reason":null,"delta":{"content":"ld!"}}],"created":1764692529} + + id: 4 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[{"index":0,"finish_reason":"stop","delta":{"content":""}}],"created":1764692529} + + id: 5 + event: message + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model_id":"ibm/granite-4-h-small","model":"ibm/granite-4-h-small","choices":[],"created":1764692529,"usage":{"completion_tokens":3,"prompt_tokens":10,"total_tokens":13}} + """; @Test void should_stream_chat_response_in_chunks_correctly() throws Exception { @@ -1569,7 +1594,7 @@ public void onPartialThinking(String partialThinking, PartialChatResponse partia } @Test - void should_not_override_assistant_content_in_streaming() { + void should_apply_message_interceptor_to_the_complete_streaming_message() { String REQUEST = """ { @@ -1694,7 +1719,7 @@ void should_not_override_assistant_content_in_streaming() { CompletableFuture result = new CompletableFuture<>(); List partialResponses = new ArrayList<>(); - chatService.chatStreaming("How are you?", new ChatHandler() { + var future = chatService.chatStreaming("How are you?", new ChatHandler() { @Override public void onPartialResponse(String partialResponse, PartialChatResponse partialChatResponse) { @@ -1713,10 +1738,15 @@ public void onError(Throwable error) { }); var chatResponse = assertDoesNotThrow(() -> result.get(3, TimeUnit.SECONDS)); + var returnedResponse = assertDoesNotThrow(() -> future.get(3, TimeUnit.SECONDS)); var assistantMessage = chatResponse.toAssistantMessage(); - assertEquals("Hello! I'm doing well, thank you. How can I assist you today?", assistantMessage.content()); + + assertEquals("I don't feel good.", assistantMessage.content()); + assertEquals("I don't feel good.", returnedResponse.toAssistantMessage().content()); assertFalse(assistantMessage.hasToolCalls()); - assertEquals("Hello! I'm doing well, thank you. How can I assist you today?", partialResponses.stream().collect(Collectors.joining())); + assertEquals( + "Hello! I'm doing well, thank you. How can I assist you today?", + partialResponses.stream().collect(Collectors.joining())); } @Test @@ -4277,4 +4307,232 @@ void should_complete_the_stream_when_the_moderation_system_reports_the_output_wi assertFalse(textChatResponse.moderations().get("pii").get(0).input()); assertEquals("Call 3334523123", assertDoesNotThrow(chatResponse::toAssistantMessage).content()); } + + @Test + void should_apply_message_interceptor_across_token_boundaries_in_streaming() { + + wireMock.stubFor(post("/ml/v1/text/chat_stream?version=%s".formatted(API_VERSION)) + .willReturn(aResponse() + .withStatus(200) + .withChunkedDribbleDelay(5, 200) + .withBody(SPLIT_WORD_STREAMING_RESPONSE))); + + when(mockAuthenticator.tokenAsync()).thenReturn(completedFuture("my-super-token")); + + var intercepted = new AtomicReference(); + var chatService = ChatService.builder() + .authenticator(mockAuthenticator) + .modelId("ibm/granite-4-h-small") + .projectId("project-id") + .baseUrl(URI.create("http://localhost:%s".formatted(wireMock.getPort()))) + .messageInterceptor((ctx, message) -> { + intercepted.set(message); + return message.replace("Hello world", "Ciao mondo"); + }) + .build(); + + List partialResponses = new ArrayList<>(); + CompletableFuture delivered = new CompletableFuture<>(); + var future = chatService.chatStreaming("Hi", new ChatHandler() { + + @Override + public void onPartialResponse(String partialResponse, PartialChatResponse partialChatResponse) { + partialResponses.add(partialResponse); + } + + @Override + public void onCompleteResponse(ChatResponse completeResponse) { + delivered.complete(completeResponse); + } + + @Override + public void onError(Throwable error) { + delivered.completeExceptionally(error); + } + }); + + var returnedResponse = assertDoesNotThrow(() -> future.get(3, TimeUnit.SECONDS)); + var completeResponse = assertDoesNotThrow(() -> delivered.get(3, TimeUnit.SECONDS)); + + assertEquals("Hello world!", intercepted.get()); + assertEquals("Ciao mondo!", completeResponse.toAssistantMessage().content()); + assertEquals("Ciao mondo!", returnedResponse.toAssistantMessage().content()); + assertEquals(List.of("Hello", " wor", "ld!"), partialResponses); + } + + @Test + void should_apply_message_interceptor_to_streaming_message_with_extraction_tags() throws Exception { + + String BODY = new String(ClassLoader.getSystemResourceAsStream("granite_thinking_streaming_response.txt").readAllBytes()); + + wireMock.stubFor(post("/ml/v1/text/chat_stream?version=%s".formatted(API_VERSION)) + .willReturn(aResponse() + .withStatus(200) + .withChunkedDribbleDelay(159, 200) + .withBody(BODY))); + + when(mockAuthenticator.tokenAsync()).thenReturn(completedFuture("my-super-token")); + + var intercepted = new AtomicReference(); + var chatService = ChatService.builder() + .authenticator(mockAuthenticator) + .modelId("ibm/granite-3-3-8b-instruct") + .projectId("project-id") + .baseUrl(URI.create("http://localhost:%s".formatted(wireMock.getPort()))) + .messageInterceptor((ctx, message) -> { + intercepted.set(message); + return message.replace("Ciao", "Salve"); + }) + .build(); + + var chatRequest = ChatRequest.builder() + .messages(UserMessage.text("Translate \"Hello\" in Italian")) + .thinking(ExtractionTags.of(new Think("", ""), new Response("", ""))) + .build(); + + var partialResponses = new StringBuilder(); + CompletableFuture result = new CompletableFuture<>(); + chatService.chatStreaming(chatRequest, new ChatHandler() { + + @Override + public void onPartialResponse(String partialResponse, PartialChatResponse partialChatResponse) { + partialResponses.append(partialResponse); + } + + @Override + public void onCompleteResponse(ChatResponse completeResponse) { + result.complete(completeResponse); + } + + @Override + public void onError(Throwable error) { + result.completeExceptionally(error); + } + + @Override + public void onPartialThinking(String partialThinking, PartialChatResponse partialChatResponse) {} + }); + + var chatResponse = assertDoesNotThrow(() -> result.get(3, TimeUnit.SECONDS)); + var assistantMessage = chatResponse.toAssistantMessage(); + + assertTrue(intercepted.get().contains("") && intercepted.get().contains("")); + assertTrue(intercepted.get().contains("") && intercepted.get().contains("")); + assertFalse(assistantMessage.content().contains("Ciao")); + assertTrue(assistantMessage.content().contains("**Salve**")); + assertFalse(assistantMessage.thinking().contains("Ciao")); + assertTrue(assistantMessage.thinking().contains("\"Salve\"")); + assertTrue(partialResponses.toString().contains("**Ciao**")); + } + + @Test + void should_apply_partial_response_interceptor_in_streaming() { + + wireMock.stubFor(post("/ml/v1/text/chat_stream?version=%s".formatted(API_VERSION)) + .willReturn(aResponse() + .withStatus(200) + .withChunkedDribbleDelay(5, 200) + .withBody(SPLIT_WORD_STREAMING_RESPONSE))); + + when(mockAuthenticator.tokenAsync()).thenReturn(completedFuture("my-super-token")); + + var responseSeenByInterceptor = new AtomicBoolean(false); + var chatService = ChatService.builder() + .authenticator(mockAuthenticator) + .modelId("ibm/granite-4-h-small") + .projectId("project-id") + .baseUrl(URI.create("http://localhost:%s".formatted(wireMock.getPort()))) + .partialResponseInterceptor((ctx, partialResponse) -> { + if (ctx.response().isPresent()) + responseSeenByInterceptor.set(true); + return partialResponse.toUpperCase(); + }) + .build(); + + List partialResponses = new ArrayList<>(); + CompletableFuture delivered = new CompletableFuture<>(); + var future = chatService.chatStreaming("Hi", new ChatHandler() { + + @Override + public void onPartialResponse(String partialResponse, PartialChatResponse partialChatResponse) { + partialResponses.add(partialResponse); + } + + @Override + public void onCompleteResponse(ChatResponse completeResponse) { + delivered.complete(completeResponse); + } + + @Override + public void onError(Throwable error) { + delivered.completeExceptionally(error); + } + }); + + var returnedResponse = assertDoesNotThrow(() -> future.get(3, TimeUnit.SECONDS)); + var completeResponse = assertDoesNotThrow(() -> delivered.get(3, TimeUnit.SECONDS)); + + assertEquals(List.of("HELLO", " WOR", "LD!"), partialResponses); + assertEquals("Hello world!", completeResponse.toAssistantMessage().content()); + assertEquals("Hello world!", returnedResponse.toAssistantMessage().content()); + assertFalse(responseSeenByInterceptor.get()); + } + + @Test + void should_report_streaming_interceptor_failures_and_keep_the_original_values() { + + wireMock.stubFor(post("/ml/v1/text/chat_stream?version=%s".formatted(API_VERSION)) + .willReturn(aResponse() + .withStatus(200) + .withChunkedDribbleDelay(5, 200) + .withBody(SPLIT_WORD_STREAMING_RESPONSE))); + + when(mockAuthenticator.tokenAsync()).thenReturn(completedFuture("my-super-token")); + + var chatService = ChatService.builder() + .authenticator(mockAuthenticator) + .modelId("ibm/granite-4-h-small") + .projectId("project-id") + .baseUrl(URI.create("http://localhost:%s".formatted(wireMock.getPort()))) + .messageInterceptor((ctx, message) -> { + throw new IllegalStateException("message boom"); + }) + .partialResponseInterceptor((ctx, partialResponse) -> { + throw new IllegalStateException("partial boom"); + }) + .build(); + + List partialResponses = Collections.synchronizedList(new ArrayList<>()); + List errors = Collections.synchronizedList(new ArrayList<>()); + CompletableFuture delivered = new CompletableFuture<>(); + var future = chatService.chatStreaming("Hi", new ChatHandler() { + + @Override + public void onPartialResponse(String partialResponse, PartialChatResponse partialChatResponse) { + partialResponses.add(partialResponse); + } + + @Override + public void onCompleteResponse(ChatResponse completeResponse) { + delivered.complete(completeResponse); + } + + @Override + public void onError(Throwable error) { + errors.add(error); + } + }); + + var returnedResponse = assertDoesNotThrow(() -> future.get(3, TimeUnit.SECONDS)); + var completeResponse = assertDoesNotThrow(() -> delivered.get(3, TimeUnit.SECONDS)); + + assertEquals(List.of("Hello", " wor", "ld!"), partialResponses); + assertEquals("Hello world!", completeResponse.toAssistantMessage().content()); + assertEquals("Hello world!", returnedResponse.toAssistantMessage().content()); + + var messages = errors.stream().map(Throwable::getMessage).collect(Collectors.toList()); + assertEquals(4, messages.size()); + assertEquals(3, Collections.frequency(messages, "partial boom")); + assertEquals(1, Collections.frequency(messages, "message boom")); + } } diff --git a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/ChatStreamingCancellationTest.java b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/ChatStreamingCancellationTest.java index 3d951f4d..50306177 100644 --- a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/ChatStreamingCancellationTest.java +++ b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/ChatStreamingCancellationTest.java @@ -417,7 +417,7 @@ void should_treat_cancel_false_like_cancel_true() throws Exception { void should_cancel_the_body_subscription() { var recorder = new Recorder(0); - var decorator = new ChatHandlerDecorator(recorder, null, null); + var decorator = new ChatHandlerDecorator(recorder, null, null, null, null); var subscriber = new DefaultChatSubscriber(new SseEventProcessor(null, null, TextChatResponse::builder), decorator); var response = cancellableResponse(subscriber); var flowSubscriber = subscriber.asFlowSubscriber(response, true); @@ -444,7 +444,7 @@ void should_cancel_the_body_subscription() { void should_drop_the_signals_delivered_after_cancel() { var recorder = new Recorder(0); - var decorator = new ChatHandlerDecorator(recorder, null, null); + var decorator = new ChatHandlerDecorator(recorder, null, null, null, null); var subscriber = new DefaultChatSubscriber(new SseEventProcessor(null, null, TextChatResponse::builder), decorator); var response = cancellableResponse(subscriber); var flowSubscriber = subscriber.asFlowSubscriber(response, true); @@ -474,7 +474,7 @@ void should_drop_the_signals_delivered_after_cancel() { void should_not_report_a_moderation_error_when_the_stream_is_cancelled() { var recorder = new Recorder(0); - var decorator = new ChatHandlerDecorator(recorder, null, null); + var decorator = new ChatHandlerDecorator(recorder, null, null, null, null); var subscriber = new DefaultChatSubscriber(new SseEventProcessor(null, null, TextChatResponse::builder), decorator); var response = cancellableResponse(subscriber); var flowSubscriber = subscriber.asFlowSubscriber(response, true); diff --git a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/decorator/ChatHandlerDecoratorTest.java b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/decorator/ChatHandlerDecoratorTest.java index b7bf830b..8dd83938 100644 --- a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/decorator/ChatHandlerDecoratorTest.java +++ b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/chat/decorator/ChatHandlerDecoratorTest.java @@ -93,7 +93,7 @@ static void emitTwoToolCalls(ChatHandlerDecorator decorator) { void should_deliver_every_callback_in_emission_order() { var recorder = new Recorder(); - var decorator = new ChatHandlerDecorator(recorder, null, null); + var decorator = new ChatHandlerDecorator(recorder, null, null, null, null); emitTwoToolCalls(decorator); decorator.awaitCallbacks().join(); @@ -110,7 +110,7 @@ void should_deliver_every_callback_in_emission_order_with_a_slow_interceptor() { return functionCall; }; - var decorator = new ChatHandlerDecorator(recorder, null, slow); + var decorator = new ChatHandlerDecorator(recorder, null, null, null, slow); emitTwoToolCalls(decorator); decorator.awaitCallbacks().join(); @@ -151,7 +151,7 @@ public void onCompleteToolCall(CompletedToolCall completeToolCall) { public void onCompleteResponse(ChatResponse completeResponse) { body.run(); } - }, null, null); + }, null, null, null, null); emitTwoToolCalls(decorator); decorator.awaitCallbacks().join(); @@ -185,7 +185,7 @@ public void onCompleteResponse(ChatResponse completeResponse) { public void onError(Throwable error) { recorder.onError(error); } - }, null, null); + }, null, null, null, null); decorator.onPartialResponse("Hello", null); decorator.onPartialToolCall(partial(0, "get_weather")); @@ -216,7 +216,7 @@ public void onCompleteToolCall(CompletedToolCall completeToolCall) { public void onError(Throwable error) { recorder.onError(error); } - }, null, null); + }, null, null, null, null); decorator.onCompleteToolCall(complete(0, "get_weather")); decorator.onPartialResponse("Hello", null); @@ -236,7 +236,7 @@ void should_report_interceptor_failure_and_deliver_the_un_normalized_tool_call() throw new IllegalStateException("thrown by the interceptor"); }; - var decorator = new ChatHandlerDecorator(recorder, null, failing); + var decorator = new ChatHandlerDecorator(recorder, null, null, null, failing); decorator.onPartialToolCall(partial(0, "get_weather")); decorator.onCompleteToolCall(complete(0, "get_weather")); @@ -252,7 +252,7 @@ void should_report_interceptor_failure_and_deliver_the_un_normalized_tool_call() @Test void should_return_tool_calls_in_receive_order() { - var decorator = new ChatHandlerDecorator(new Recorder(), null, null); + var decorator = new ChatHandlerDecorator(new Recorder(), null, null, null, null); decorator.onCompleteToolCall(complete(0, "get_weather")); decorator.onCompleteToolCall(complete(1, "get_current_time")); @@ -264,7 +264,7 @@ void should_return_tool_calls_in_receive_order() { @Test void should_take_effect_only_once_when_cancelled_more_than_once() { - var decorator = new ChatHandlerDecorator(new Recorder(), null, null); + var decorator = new ChatHandlerDecorator(new Recorder(), null, null, null, null); assertTrue(decorator.cancel()); assertFalse(decorator.cancel()); @@ -275,7 +275,7 @@ void should_take_effect_only_once_when_cancelled_more_than_once() { void should_deliver_no_callback_after_cancel() { var recorder = new Recorder(); - var decorator = new ChatHandlerDecorator(recorder, null, null); + var decorator = new ChatHandlerDecorator(recorder, null, null, null, null); decorator.onPartialResponse("Hello", null); decorator.awaitCallbacks().join(); @@ -313,7 +313,7 @@ public void onCompleteResponse(ChatResponse completeResponse) { public void onError(Throwable error) { recorder.onError(error); } - }, null, null)); + }, null, null, null, null)); decorator.get().onPartialResponse("Hello", null); decorator.get().onCompleteResponse(null); diff --git a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/gateway/chat/ModelGatewayChatServiceTest.java b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/gateway/chat/ModelGatewayChatServiceTest.java index 8ffeb387..66c3192d 100644 --- a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/gateway/chat/ModelGatewayChatServiceTest.java +++ b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/gateway/chat/ModelGatewayChatServiceTest.java @@ -31,6 +31,7 @@ import java.util.Map; import java.util.concurrent.CompletableFuture; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; import java.util.function.Consumer; import org.junit.jupiter.api.Test; import org.skyscreamer.jsonassert.JSONAssert; @@ -981,4 +982,68 @@ void should_throw_exception_when_using_control_message() { assertEquals("Control messages are not supported by the Model Gateway", ex.getMessage()); }); } + + @Test + void should_apply_message_and_partial_response_interceptors_in_streaming() { + + wireMock.stubFor(post("/ml/gateway/v1/chat/completions?version=%s".formatted(API_VERSION)) + .willReturn(aResponse() + .withStatus(200) + .withChunkedDribbleDelay(5, 200) + .withBody( + """ + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"},"finish_reason":""}],"created":1785169730,"model":"gpt-4o"} + + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" wor"},"finish_reason":""}],"created":1785169730,"model":"gpt-4o"} + + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"ld!"},"finish_reason":""}],"created":1785169730,"model":"gpt-4o"} + + data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":"stop"}],"created":1785169730,"model":"gpt-4o"} + + data: [DONE] + """))); + + when(mockAuthenticator.tokenAsync()).thenReturn(completedFuture("my-super-token")); + + var intercepted = new AtomicReference(); + var modelGatewayChatService = ModelGatewayChatService.builder() + .authenticator(mockAuthenticator) + .modelId("gpt-4o") + .baseUrl(URI.create("http://localhost:%s".formatted(wireMock.getPort()))) + .version(API_VERSION) + .messageInterceptor((ctx, message) -> { + intercepted.set(message); + return message.replace("Hello world", "Ciao mondo"); + }) + .partialResponseInterceptor((ctx, partialResponse) -> partialResponse.toUpperCase()) + .build(); + + List partialResponses = new ArrayList<>(); + CompletableFuture delivered = new CompletableFuture<>(); + var future = modelGatewayChatService.chatStreaming("Hi", new ChatHandler() { + + @Override + public void onPartialResponse(String partialResponse, PartialChatResponse partialChatResponse) { + partialResponses.add(partialResponse); + } + + @Override + public void onCompleteResponse(ChatResponse completeResponse) { + delivered.complete(completeResponse); + } + + @Override + public void onError(Throwable error) { + delivered.completeExceptionally(error); + } + }); + + var returnedResponse = assertDoesNotThrow(() -> future.get(3, TimeUnit.SECONDS)); + var completeResponse = assertDoesNotThrow(() -> delivered.get(3, TimeUnit.SECONDS)); + + assertEquals("Hello world!", intercepted.get()); + assertEquals(List.of("HELLO", " WOR", "LD!"), partialResponses); + assertEquals("Ciao mondo!", completeResponse.toAssistantMessage().content()); + assertEquals("Ciao mondo!", returnedResponse.toAssistantMessage().content()); + } } diff --git a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/it/ModelGatewayChatServiceIT.java b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/it/ModelGatewayChatServiceIT.java index 26001ee2..8d56b6a0 100644 --- a/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/it/ModelGatewayChatServiceIT.java +++ b/modules/watsonx-ai/src/test/java/com/ibm/watsonx/ai/it/ModelGatewayChatServiceIT.java @@ -37,6 +37,7 @@ import com.google.common.collect.Sets; import com.ibm.watsonx.ai.chat.ChatHandler; import com.ibm.watsonx.ai.chat.ChatResponse; +import com.ibm.watsonx.ai.chat.interceptor.MessageInterceptor; import com.ibm.watsonx.ai.chat.model.AssistantMessage; import com.ibm.watsonx.ai.chat.model.BaseChatParameters.ToolChoiceOption; import com.ibm.watsonx.ai.chat.model.ChatMessage; @@ -64,14 +65,19 @@ @EnabledIfEnvironmentVariable(named = "WATSONX_API_KEY", matches = ".+") @EnabledIfEnvironmentVariable(named = "WATSONX_URL", matches = ".+") @EnabledIfEnvironmentVariable(named = "WATSONX_GATEWAY_CHAT_MODEL_CLAUDE", matches = ".+") +@EnabledIfEnvironmentVariable(named = "WATSONX_GATEWAY_CHAT_MODEL_GEMMA", matches = ".+") @EnabledIfEnvironmentVariable(named = "WATSONX_GATEWAY_CHAT_MODEL_OPENAI", matches = ".+") public class ModelGatewayChatServiceIT { static final String API_KEY = System.getenv("WATSONX_API_KEY"); static final String URL = System.getenv("WATSONX_URL"); static final String CHAT_MODEL_CLAUDE = System.getenv("WATSONX_GATEWAY_CHAT_MODEL_CLAUDE"); + static final String CHAT_MODEL_GEMMA = System.getenv("WATSONX_GATEWAY_CHAT_MODEL_GEMMA"); static final String CHAT_MODEL_OPENAI = System.getenv("WATSONX_GATEWAY_CHAT_MODEL_OPENAI"); + static final MessageInterceptor SANITIZE_TOOL = + (ctx, message) -> message.replace("```json", "").replace("```", ""); + static final Authenticator authentication = IBMCloudAuthenticator.builder() .apiKey(API_KEY) .build(); @@ -153,7 +159,7 @@ void should_follow_the_instruction_when_developer_message_is_sent() { var modelGatewayChatService = ModelGatewayChatService.builder() .baseUrl(URL) - .modelId(CHAT_MODEL_OPENAI) + .modelId(CHAT_MODEL_CLAUDE) .authenticator(authentication) .logRequests(true) .logResponses(true) @@ -183,6 +189,7 @@ record Poem(String content, String topic) {} .baseUrl(URL) .modelId(CHAT_MODEL_CLAUDE) .authenticator(authentication) + .messageInterceptor(SANITIZE_TOOL) .logRequests(true) .logResponses(true) .build(); @@ -219,7 +226,7 @@ record Poem(String content, String topic) {} var modelGatewayChatService = ModelGatewayChatService.builder() .baseUrl(URL) - .modelId(CHAT_MODEL_OPENAI) + .modelId(CHAT_MODEL_CLAUDE) .authenticator(authentication) .logRequests(true) .logResponses(true) @@ -232,7 +239,6 @@ record Poem(String content, String topic) {} .property("content", JsonSchema.string()) .property("topic", JsonSchema.enumeration("dog", "cat")) .required("content", "topic") - .additionalProperties(false) .build(), true) .build(); @@ -376,7 +382,7 @@ void should_force_tool_execution_when_tool_choice_option_is_set_to_required() { var modelGatewayChatService = ModelGatewayChatService.builder() .baseUrl(URL) - .modelId(CHAT_MODEL_OPENAI) + .modelId(CHAT_MODEL_GEMMA) .authenticator(authentication) .logRequests(true) .logResponses(true) @@ -410,7 +416,7 @@ void should_not_force_tool_execution_when_tool_choice_option_is_set_to_none() { var modelGatewayChatService = ModelGatewayChatService.builder() .baseUrl(URL) - .modelId(CHAT_MODEL_OPENAI) + .modelId(CHAT_MODEL_GEMMA) .authenticator(authentication) .logRequests(true) .logResponses(true) @@ -678,6 +684,7 @@ record Poem(String content, String topic) {} .baseUrl(URL) .modelId(CHAT_MODEL_CLAUDE) .authenticator(authentication) + .messageInterceptor(SANITIZE_TOOL) .logRequests(true) .logResponses(true) .build(); @@ -713,7 +720,7 @@ public void onCompleteResponse(ChatResponse completeResponse) { public void onError(Throwable error) {} }); - var chatResponse = assertDoesNotThrow(() -> future.get(5, TimeUnit.SECONDS)); + var chatResponse = assertDoesNotThrow(() -> future.get(500, TimeUnit.SECONDS)); var poem = chatResponse.toAssistantMessage().toObject(Poem.class); assertNotNull(chatResponse); @@ -1121,7 +1128,7 @@ void should_force_tool_execution_when_tool_choice_option_is_set_to_required() { var modelGatewayChatService = ModelGatewayChatService.builder() .baseUrl(URL) - .modelId(CHAT_MODEL_CLAUDE) + .modelId(CHAT_MODEL_GEMMA) .authenticator(authentication) .logRequests(true) .logResponses(true) @@ -1164,7 +1171,7 @@ public void onCompleteToolCall(CompletedToolCall completeToolCall) { public void onError(Throwable error) {} })); - var chatResponse = assertDoesNotThrow(() -> future.get(30, TimeUnit.SECONDS)); + var chatResponse = assertDoesNotThrow(() -> future.get(60, TimeUnit.SECONDS)); var completedToolCall = assertDoesNotThrow(() -> futureToolCall.get(10, TimeUnit.SECONDS)); var assistantMessage = chatResponse.toAssistantMessage(); assertTrue(assistantMessage.content() == null || assistantMessage.content().isBlank());