improvement(search): bounded Jira fallback, organization search through integrations page connections (#8985)

* improvement(search): bound the any-term fallback so a slow relaxed query keeps the strict page

When an all-terms Atlassian search underfills, the live search runs a relaxed any-term query.
For Jira that query ORs every pair of up to 8 terms (28 pairs, 56 `text ~` clauses), which can
run past the 10 s HTTP timeout on a large site. The transport failure was rethrown, so the whole
account leg reported unavailable: the strict matches already fetched were discarded and the
search waited out the timeout.

- The relaxed search gets its own 4 s budget, composed with the caller's signal through
  AbortSignal.any and passed down as a per-search signal (NativeSearchInput.signal ->
  searchAtlassian -> client.json -> secureFetchWithValidation), so the HTTP request itself is
  cancelled rather than raced. The account leg already has a 12 s deadline that covers resolve,
  session, the strict search, verification, and the 2.5 s preview reads; a relaxed query
  normally takes ~1.5 s, so 4 s leaves headroom for a slow but healthy query while ending a
  hung one with time left for verification and previews inside the leg deadline. Composing with
  the leg signal means it can never run past that deadline.
- Any relaxed failure other than the caller's own cancellation (provider refusal, the budget
  running out, a transport error) keeps the strict page, marked partial with a message, the same
  shape as the existing refusal path. Caller cancellation still propagates. A successful relaxed
  search behaves exactly as before.
- Jira relaxed JQL pairs at most the six longest distinct terms (at most 15 pairs), and a word
  and its regular plural share one slot, since Jira stems text search. Ranking by term coverage
  still uses every significant term, and Confluence's siteSearch clause is unchanged.
- A memoized GET is never shared with a request that carries its own signal.

* improvement(search): keep Jira's eight-term pairing, simplify fallback error handling

Drop the six-term pair cap (the budget already bounds a slow relaxed query, and the cap removed pairs of the shortest terms). Keep the singular/plural collapse, now leaving -us words (status, focus) whole. One branch per relaxed failure kind, the term cap inside distinctTerms, and no memo special case no caller reaches.

* fix(search): search an organization only through its integrations page connections

Organization live search also listed a member's plain OAuth credentials at organization scope, which no page shows or can disconnect. One left behind next to the same person's integrations page connection made every result from that provider come back twice. Organization search now uses only credential-group enrollments; workspace search is unchanged.

Author: waleedlatif1

apps/sim/lib/sim-search/live/account-session.ts

@@ -206,7 +206,7 @@ export async function openLiveAccountSession(
         )
       }
       return client && requiresEveryTerm(provider)
-        ? searchWithAnyTermFallback(searchOnce, search)
+        ? searchWithAnyTermFallback(searchOnce, search, signal)
         : searchOnce(search)
     },
     async verify(document) {

apps/sim/lib/sim-search/live/accounts.test.ts

@@ -130,6 +130,25 @@ describe('live account discovery boundaries', () => {
     })
     expect(mocks.resolve).not.toHaveBeenCalled()
   })
+  it('searches an organization only through its integrations page enrollments', async () => {
+    mockListOrganizationSearchApprovals.mockResolvedValue(new Map([['jira', true]]))
+    mocks.managed.mockResolvedValue([
+      { id: 'enrolled', providerId: 'jira', displayName: 'Enrolled Jira' },
+    ])
+    const legacy = { id: 'legacy', providerId: 'jira', displayName: 'Legacy Jira' }
+    const metadata = ['enrolled', 'legacy'].map((id) => ({
+      id,
+      scope: 'read:jira-work',
+      grantedScopes: null,
+      revokedAt: null,
+    }))
+    queueTableRows(schemaMock.credential, [legacy])
+    queueTableRows(schemaMock.credential, metadata)
+    queueTableRows(schemaMock.credential, metadata)
+    expect(
+      (await listLiveAccounts({ organizationId: 'org' }, 'reader')).map((row) => row.id)
+    ).toEqual(['enrolled'])
+  })
   it('does not resolve another account even when the caller knows its ID', async () => {
     metadata()
     await expect(resolveLiveAccount(owner, 'reader', 'someone-else')).rejects.toMatchObject({

apps/sim/lib/sim-search/live/accounts.ts

@@ -1,6 +1,6 @@
 import { db } from '@sim/db'
 import { account, credential, user } from '@sim/db/schema'
-import { and, eq, inArray, isNull, ne } from 'drizzle-orm'
+import { and, eq, inArray, isNull } from 'drizzle-orm'
 import { liveSearchProviderSchema } from '@/lib/api/contracts/mothership-assistant-tools'
 import {
   type ResourceOwner,
@@ -28,34 +28,17 @@ import {
 } from '@/lib/sim-search/live/provider-catalog'
 import type { LiveAccount } from '@/lib/sim-search/live/types'
 
+/**
+ * A workspace searches the person's workspace credentials; an organization searches only the
+ * accounts its members connect on the integrations page (credential-group enrollments).
+ */
 async function listOAuthRows(scope: ResourceScope, userId: string) {
   if (scope.kind === 'workspace') return getPersonalOAuthCredentials(scope.workspaceId, userId)
-  const [managed, personal] = await Promise.all([
-    getOwnOrganizationManagedOAuthCredentials({ organizationId: scope.organizationId, userId }),
-    db
-      .select({
-        id: credential.id,
-        providerId: credential.providerId,
-        displayName: credential.displayName,
-      })
-      .from(credential)
-      .innerJoin(account, eq(account.id, credential.accountId))
-      .where(
-        and(
-          resourceScopeCondition(credential, scope),
-          eq(credential.type, 'oauth'),
-          eq(account.userId, userId),
-          eq(account.providerId, credential.providerId),
-          ne(credential.providerId, 'slack')
-        )
-      ),
-  ])
-  return [
-    ...managed.map((row) => ({ ...row, type: 'managed_oauth' as const })),
-    ...personal.flatMap((row) =>
-      row.providerId ? [{ ...row, providerId: row.providerId, type: 'oauth' as const }] : []
-    ),
-  ]
+  const managed = await getOwnOrganizationManagedOAuthCredentials({
+    organizationId: scope.organizationId,
+    userId,
+  })
+  return managed.map((row) => ({ ...row, type: 'managed_oauth' as const }))
 }
 
 async function listCodaTokenRows(scope: ResourceScope, userId: string) {

apps/sim/lib/sim-search/live/atlassian.ts

@@ -169,6 +169,7 @@ export async function searchAtlassian(
               fields: ['summary', 'description', 'updated', 'creator', 'status'],
               ...(input.native?.cursor && single ? { nextPageToken: input.native.cursor } : {}),
             },
+            signal: input.signal,
           })
         )
         return {
@@ -190,6 +191,7 @@ export async function searchAtlassian(
             expand: 'content.version,content.space',
             ...(input.native?.cursor && single ? { cursor: input.native.cursor } : {}),
           },
+          signal: input.signal,
         })
       )
       const next = string(object(data._links).next)

apps/sim/lib/sim-search/live/http.ts

@@ -39,9 +39,11 @@ export function createNativeClient(input: {
       query?: Record<string, string | string[]>
       body?: unknown
       googleService?: 'sheets'
+      signal?: AbortSignal
     }
   ) {
-    input.signal.throwIfAborted()
+    const signal = options?.signal ? AbortSignal.any([input.signal, options.signal]) : input.signal
+    signal.throwIfAborted()
     if (++requests > (input.requestBudget ?? NATIVE_SEARCH_REQUEST_BUDGET))
       throw new NativeSearchError('unavailable', 'Request budget reached. Narrow the query.')
     const url = new URL(input.origin)
@@ -68,7 +70,7 @@ export function createNativeClient(input: {
       },
       method: options?.body ? 'POST' : 'GET',
       ...(options?.body ? { body: JSON.stringify(options.body) } : {}),
-      signal: input.signal,
+      signal,
       timeout: 10_000,
       maxResponseBytes: NATIVE_RESPONSE_MAX_BYTES,
       maxRedirects: 0,

apps/sim/lib/sim-search/live/term-fallback.test.ts

@@ -1,10 +1,30 @@
-import { describe, expect, it, vi } from 'vitest'
-import { NativeSearchError } from '@/lib/sim-search/live/http'
+import {
+  inputValidationMock,
+  inputValidationMockFns,
+} from '@sim/testing/mocks/input-validation.mock'
+import { afterEach, describe, expect, it, vi } from 'vitest'
+
+vi.mock('@/lib/core/security/input-validation.server', () => inputValidationMock)
+
+import { searchAtlassian } from '@/lib/sim-search/live/atlassian'
+import { createNativeClient, NativeSearchError } from '@/lib/sim-search/live/http'
 import { searchWithAnyTermFallback } from '@/lib/sim-search/live/term-fallback'
 import type { NativeDocument, NativePage, NativeSearchInput } from '@/lib/sim-search/live/types'
 
 const input: NativeSearchInput = { query: 'contractor offboarding runbook', limit: 4, scopes: [] }
 
+/** The any-term search's own budget, which a slow provider must not outlast. */
+const BUDGET_MS = 4000
+
+/** A caller that never cancels. */
+const live = new AbortController().signal
+
+/** Settles only when its request is cancelled, like a provider that never answers. */
+const untilAborted = (signal: AbortSignal | undefined) =>
+  new Promise<never>((_, reject) => {
+    signal?.addEventListener('abort', () => reject(signal.reason), { once: true })
+  })
+
 function page(...documents: Partial<NativeDocument>[]): NativePage {
   return {
     documents: documents.map((document, index) => ({
@@ -26,6 +46,10 @@ function provider(strict: NativePage, relaxed: NativePage | Error) {
 }
 
 describe('searchWithAnyTermFallback', () => {
+  afterEach(() => {
+    vi.useRealTimers()
+  })
+
   it('follows an exhausted all-terms page with partial matches, most terms first', async () => {
     const search = provider(
       page({ id: 'exact', title: 'Contractor offboarding runbook' }),
@@ -37,14 +61,15 @@ describe('searchWithAnyTermFallback', () => {
         { id: 'contractors', title: 'Contractor directory' }
       )
     )
-    const result = await searchWithAnyTermFallback(search, input)
+    const result = await searchWithAnyTermFallback(search, input, live)
     expect(result.documents.map((document) => document.id)).toEqual([
       'exact',
       'in-title',
       'in-preview',
       'runbook',
     ])
     expect(result.nextCursor).toBeUndefined()
+    expect(result.partial).toBeFalsy()
     expect(result.message).toContain('Few results contained every search term')
   })
 
@@ -58,7 +83,7 @@ describe('searchWithAnyTermFallback', () => {
         ? { documents: page(...loose).documents.slice(0, request.limit) }
         : page()
     )
-    const result = await searchWithAnyTermFallback(search, { ...input, limit: 2 })
+    const result = await searchWithAnyTermFallback(search, { ...input, limit: 2 }, live)
     expect(result.documents.map((document) => document.title)).toEqual([
       'Contractor offboarding form',
       'Runbook 0',
@@ -68,34 +93,36 @@ describe('searchWithAnyTermFallback', () => {
   it('adds a few partial matches however large the page', async () => {
     const partial = Array.from({ length: 12 }, () => ({ title: 'Contractor offboarding' }))
     const search = provider(page(), page(...partial))
-    const result = await searchWithAnyTermFallback(search, { ...input, limit: 20 })
+    const result = await searchWithAnyTermFallback(search, { ...input, limit: 20 }, live)
     expect(result.documents).toHaveLength(5)
   })
 
   it('never relaxes a native query, which is the caller’s exact syntax', async () => {
     const search = provider(page(), page({ id: 'loose' }))
     const native = { provider: 'confluence' as const, query: 'text ~ "runbook"' }
-    expect((await searchWithAnyTermFallback(search, { ...input, native })).documents).toEqual([])
+    expect((await searchWithAnyTermFallback(search, { ...input, native }, live)).documents).toEqual(
+      []
+    )
     expect(search).toHaveBeenCalledTimes(1)
   })
 
   it('keeps a continuable page so its cursor continues the query that produced it', async () => {
     const strict = { ...page({ id: 'exact' }), nextCursor: 'page-2' }
     const search = provider(strict, page({ id: 'loose' }))
-    expect(await searchWithAnyTermFallback(search, input)).toBe(strict)
+    expect(await searchWithAnyTermFallback(search, input, live)).toBe(strict)
     expect(search).toHaveBeenCalledTimes(1)
   })
 
   it('does not relax a page that is already full', async () => {
     const strict = page({}, {}, {}, {})
     const search = provider(strict, page({ id: 'loose' }))
-    expect(await searchWithAnyTermFallback(search, input)).toBe(strict)
+    expect(await searchWithAnyTermFallback(search, input, live)).toBe(strict)
     expect(search).toHaveBeenCalledTimes(1)
   })
 
   it('does not relax a query with fewer than two distinctive terms', async () => {
     const search = provider(page(), page({ id: 'loose' }))
-    await searchWithAnyTermFallback(search, { ...input, query: 'what is the runbook' })
+    await searchWithAnyTermFallback(search, { ...input, query: 'what is the runbook' }, live)
     expect(search).toHaveBeenCalledTimes(1)
   })
 
@@ -104,25 +131,95 @@ describe('searchWithAnyTermFallback', () => {
       page({ id: 'exact' }),
       new NativeSearchError('rate_limited', 'Slow down')
     )
-    const result = await searchWithAnyTermFallback(search, input)
+    const result = await searchWithAnyTermFallback(search, input, live)
     expect(result.documents.map((document) => document.id)).toEqual(['exact'])
     expect(result).toMatchObject({ partial: true, message: 'Slow down' })
   })
 
-  it('propagates cancellation during the any-term search instead of returning a page', async () => {
-    const search = provider(page({ id: 'exact' }), new DOMException('Timed out', 'TimeoutError'))
-    await expect(searchWithAnyTermFallback(search, input)).rejects.toThrow('Timed out')
+  it('propagates the caller’s cancellation during the any-term search instead of returning a page', async () => {
+    const caller = new AbortController()
+    const search = vi.fn(async (request: NativeSearchInput) =>
+      request.textMatch === 'any' ? untilAborted(request.signal) : page({ id: 'exact' })
+    )
+    const result = searchWithAnyTermFallback(search, input, caller.signal)
+    await vi.waitFor(() => expect(search).toHaveBeenCalledTimes(2))
+    caller.abort(new DOMException('Cancelled', 'AbortError'))
+    await expect(result).rejects.toThrow('Cancelled')
+  })
+
+  it('keeps the all-terms page as partial when the any-term search outlasts its budget', async () => {
+    vi.useFakeTimers()
+    let relaxedSignal: AbortSignal | undefined
+    const search = vi.fn(async (request: NativeSearchInput) => {
+      if (request.textMatch !== 'any') return page({ id: 'exact' })
+      relaxedSignal = request.signal
+      return untilAborted(request.signal)
+    })
+    let settled = false
+    const result = searchWithAnyTermFallback(search, input, live).finally(() => {
+      settled = true
+    })
+    await vi.advanceTimersByTimeAsync(BUDGET_MS - 1)
+    expect(settled).toBe(false)
+    await vi.advanceTimersByTimeAsync(1)
+    const searched = await result
+    expect(relaxedSignal?.aborted).toBe(true)
+    expect(searched.documents.map((document) => document.id)).toEqual(['exact'])
+    expect(searched).toMatchObject({ partial: true })
+    expect(searched.message).toContain('some of the search terms')
+  })
+
+  it('keeps the all-terms page as partial when the any-term search fails in transport', async () => {
+    const search = provider(page({ id: 'exact' }), new TypeError('fetch failed'))
+    const searched = await searchWithAnyTermFallback(search, input, live)
+    expect(searched.documents.map((document) => document.id)).toEqual(['exact'])
+    expect(searched).toMatchObject({ partial: true })
+    expect(searched.message).toContain('some of the search terms')
+  })
+
+  it('cancels a slow relaxed Jira request at the network once the budget runs out', async () => {
+    vi.useFakeTimers()
+    const relaxedSignals: AbortSignal[] = []
+    inputValidationMockFns.mockSecureFetchWithValidation.mockImplementation(
+      async (url: string, init: { body?: string; signal: AbortSignal }) => {
+        if (url.endsWith('/oauth/token/accessible-resources'))
+          return Response.json([{ id: 'cloud', url: 'https://acme.atlassian.net' }])
+        const { jql } = JSON.parse(init.body ?? '{}') as { jql: string }
+        if (!jql.includes(' OR '))
+          return Response.json({ issues: [{ key: 'ENG-1', fields: { summary: 'Exact' } }] })
+        relaxedSignals.push(init.signal)
+        return untilAborted(init.signal)
+      }
+    )
+    const caller = new AbortController()
+    const client = createNativeClient({
+      origin: 'https://api.atlassian.com',
+      accessToken: 'private',
+      signal: caller.signal,
+    })
+    const result = searchWithAnyTermFallback(
+      (request) => searchAtlassian(client, 'jira', request),
+      input,
+      caller.signal
+    )
+    await vi.advanceTimersByTimeAsync(BUDGET_MS)
+    const searched = await result
+    expect(relaxedSignals).toHaveLength(1)
+    expect(relaxedSignals[0]?.aborted).toBe(true)
+    expect(caller.signal.aborted).toBe(false)
+    expect(searched.documents.map((document) => document.id)).toEqual(['ENG-1'])
+    expect(searched.partial).toBe(true)
   })
 
   it('does not relax a date-sorted search, whose page is reordered by date', async () => {
     const search = provider(page(), page({ id: 'loose' }))
-    await searchWithAnyTermFallback(search, { ...input, filters: { sortBy: 'newest' } })
+    await searchWithAnyTermFallback(search, { ...input, filters: { sortBy: 'newest' } }, live)
     expect(search).toHaveBeenCalledTimes(1)
   })
 
   it('leaves the page untouched when the any-term search adds nothing new', async () => {
     const strict = page({ id: 'exact' })
     const search = provider(strict, page({ id: 'exact' }))
-    expect(await searchWithAnyTermFallback(search, input)).toBe(strict)
+    expect(await searchWithAnyTermFallback(search, input, live)).toBe(strict)
   })
 })