Skip to content

Optimize related query response latency post resolution search #785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ngoiyaeric wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Optimize related query response latency post resolution search #785

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions app/actions.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,8 +244,6 @@ async function submit(formData?: FormData, skip?: boolean) {
</Section>
);

await new Promise(resolve => setTimeout(resolve, 500));

aiState.done({
...aiState.get(),
messages: [
Expand Down Expand Up @@ -579,8 +577,6 @@ async function submit(formData?: FormData, skip?: boolean) {
</Section>
)

await new Promise(resolve => setTimeout(resolve, 500))

aiState.done({
...aiState.get(),
messages: [
Expand Down
18 changes: 11 additions & 7 deletions lib/agents/query-suggestor.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,11 @@ function getCacheKey(messages: CoreMessage[]): string {
const recentMessages = messages.slice(-3);
return JSON.stringify(recentMessages.map(m => ({
role: m.role,
content: typeof m.content === 'string' ? m.content : '[complex content]'
content: typeof m.content === 'string'
? m.content.slice(-500)

@qodo-code-review qodo-code-review Bot Sep 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Prompt suffix cache collisions 🐞 Bug ≡ Correctness

getCacheKey now retains only the last 500 characters of string messages, so different prompts
sharing that suffix receive the same cache key. During the five-minute cache lifetime,
querySuggestor can consequently return suggestions generated from another request without
consulting the model.
Agent Prompt 
## Issue description
Cache keys truncate string messages to their final 500 characters, allowing distinct model requests to share cached related-query results.

## Issue Context
The model receives the complete messages, while the module-level cache uses the truncated representation for five minutes. Generate a deterministic key from all semantically relevant message content; hash the canonical representation if bounded key size is necessary.

## Fix Focus Areas
- lib/agents/query-suggestor.tsx[18-29]
- lib/agents/query-suggestor.tsx[37-49]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

: Array.isArray(m.content)
? m.content.map((p: any) => p?.text || '').join(' ').slice(-500)
: '[complex content]'
})));
}

Expand Down Expand Up @@ -74,20 +78,20 @@ export async function querySuggestor(
return fallback
}

// OPTIMIZATION: Stream updates but batch them to reduce re-render frequency
let lastUpdateTime = Date.now();
const UPDATE_THROTTLE = 200; // ms
// OPTIMIZATION: Stream updates efficiently - update immediately on first item, then throttle
let lastUpdateTime = 0;
const UPDATE_THROTTLE = 100; // ms

try {
for await (const obj of result.partialObjectStream) {
if (obj && typeof obj === 'object' && 'items' in obj) {
finalRelatedQueries = obj as PartialRelated
const now = Date.now();
// Only update UI if enough time has passed since last update
if (now - lastUpdateTime > UPDATE_THROTTLE) {
// Update UI immediately on first yield or after throttle interval
if (lastUpdateTime === 0 || now - lastUpdateTime > UPDATE_THROTTLE) {
objectStream.update(obj as PartialRelated)
Comment on lines +90 to 92

@qodo-code-review qodo-code-review Bot Sep 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Incomplete suggestions become clickable 🐞 Bug ≡ Correctness

The first partialObjectStream value is now published immediately even though its query strings can
still be incomplete. SearchRelated enables any non-empty partial query, allowing a user to submit
truncated text before later chunks or final validation arrive.
Agent Prompt 
## Issue description
Immediately streamed partial query text is rendered as an enabled action and can be submitted before generation completes.

## Issue Context
`PartialRelated` is explicitly a deep-partial stream type. Preserve immediate visual feedback, but do not enable submission until the suggestion stream has finalized, or publish only values known to be complete.

## Fix Focus Areas
- lib/agents/query-suggestor.tsx[81-106]
- components/search-related.tsx[31-52]
- components/search-related.tsx[65-85]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

lastUpdateTime = now;
}
finalRelatedQueries = obj as PartialRelated
}
}
} catch (error) {
Expand Down