Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions skills/firecrawl-search/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ Search costs 2 credits. After you've actually used the results (or decided they

- **Time window:** must be sent within ~2 minutes of the search. Late feedback is rejected.
- **`--missing-content` is the most important field.** It's a list of _specific pieces_ of content you expected but did not find. One topic per entry, each in its own string. These aggregate across teams and tell us what to index next.
- **Include `--objective`:** the underlying goal behind the search, in one sentence — what you or your user were ultimately trying to accomplish, not only what the query looked for.
- **Substantive content required** (zero-effort feedback is rejected with HTTP 400):
- `good` → must include at least one `--valuable-sources` entry.
- `partial` → must include `--valuable-sources` or `--missing-content`.
Expand All @@ -115,6 +116,7 @@ if SEARCH_ID=$(jq -er 'select(any(.data[]; length > 0)) | .id' .firecrawl/search
--rating "<good|partial|bad>" \
--valuable-sources '[{"url":"https://react.dev/reference/react/hooks","reason":"Most authoritative"}]' \
--missing-content '[{"topic":"useDeferredValue","description":"No example of useDeferredValue with Suspense"}]' \
--objective "Pick the right React hook to keep a filtered list responsive" \
--silent &
fi
```
Expand Down
101 changes: 101 additions & 0 deletions src/__tests__/search-feedback-objective.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
import { execFile } from 'node:child_process';
import { createServer, type Server } from 'node:http';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { promisify } from 'node:util';
import { afterAll, beforeAll, beforeEach, expect, it } from 'vitest';

const exec = promisify(execFile);
const requests: { url?: string; body: any }[] = [];
let server: Server;
let baseUrl: string;
const home = mkdtempSync(join(tmpdir(), 'search-feedback-cli-'));
const searchId = '00000000-0000-4000-8000-000000000001';

beforeAll(async () => {
server = createServer(async (req, res) => {
let raw = '';
for await (const chunk of req) raw += chunk;
requests.push({ url: req.url, body: raw ? JSON.parse(raw) : undefined });
res.writeHead(200, { 'content-type': 'application/json' });
res.end(
JSON.stringify({ success: true, feedbackId: 'f-1', creditsRefunded: 1 })
);
});
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
baseUrl = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
});
afterAll(async () => {
await new Promise<void>((resolve, reject) =>
server.close((error) => (error ? reject(error) : resolve()))
);
rmSync(home, { recursive: true, force: true });
});
beforeEach(() => {
requests.length = 0;
});

async function cli(args: string[]) {
try {
return {
code: 0,
...(await exec(process.execPath, ['dist/index.js', ...args], {
timeout: 10000,
env: {
...process.env,

@cubic-dev-ai cubic-dev-ai Bot Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The child process inherits process.env, so a contributor with FIRECRAWL_NO_SEARCH_FEEDBACK or FIRECRAWL_DISABLE_SEARCH_FEEDBACK set locally gets the CLI exiting 0 without sending any request, and the first test fails on requests being empty (see isSearchFeedbackDisabledLocally in src/commands/search-feedback.ts). Neutralize both opt-out vars in the child env so the test is deterministic.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/__tests__/search-feedback-objective.test.ts, line 46:

<comment>The child process inherits `process.env`, so a contributor with `FIRECRAWL_NO_SEARCH_FEEDBACK` or `FIRECRAWL_DISABLE_SEARCH_FEEDBACK` set locally gets the CLI exiting 0 without sending any request, and the first test fails on `requests` being empty (see `isSearchFeedbackDisabledLocally` in src/commands/search-feedback.ts). Neutralize both opt-out vars in the child env so the test is deterministic.</comment>

<file context>
@@ -0,0 +1,96 @@
+      ...(await exec(process.execPath, ['dist/index.js', ...args], {
+        timeout: 10000,
+        env: {
+          ...process.env,
+          HOME: home,
+          USERPROFILE: home,
</file context>
Fix with cubic

HOME: home,
USERPROFILE: home,
FIRECRAWL_API_KEY: 'fc-test',
FIRECRAWL_API_URL: baseUrl,
FIRECRAWL_NO_UPDATE_CHECK: '1',
},
})),
};
} catch (error) {
return error as { code: number; stdout: string; stderr: string };
}
}

const feedbackArgs = [
'search-feedback',
searchId,
'--rating',
'bad',
'--missing-content',
'Contract attachments',
'--objective',
' Shortlist federal IT contracts to bid on this quarter ',
'--json',
];

it('sends the trimmed objective with search feedback', async () => {
const result = await cli(feedbackArgs);

expect(result.code).toBe(0);
expect(requests).toHaveLength(1);
expect(requests[0].url).toBe(`/v2/search/${searchId}/feedback`);
expect(requests[0].body).toMatchObject({
rating: 'bad',
objective: 'Shortlist federal IT contracts to bid on this quarter',
});
});

it('still sends feedback without an objective', async () => {

@cubic-dev-ai cubic-dev-ai Bot Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This test asserts that omitting --objective succeeds (silently no-op via --silent if the server rejects it). That contradicts the PR's stated intent of "Adds a required --objective flag," and the implementation uses a plain .option(...) (not .requiredOption), so the CLI never enforces it. If the backend now requires objective, agents that omit it (e.g. existing --silent & skills feedback) will send payloads the API rejects, and because --silent exits 0 on failure the error is swallowed. Make the flag actually required with .requiredOption(...) (and reject omission in this test), or update the PR description/skill doc to say it is optional-on-the-CLI.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/__tests__/search-feedback-objective.test.ts, line 84:

<comment>This test asserts that omitting `--objective` succeeds (silently no-op via `--silent` if the server rejects it). That contradicts the PR's stated intent of "Adds a required `--objective` flag," and the implementation uses a plain `.option(...)` (not `.requiredOption`), so the CLI never enforces it. If the backend now requires objective, agents that omit it (e.g. existing `--silent &` skills feedback) will send payloads the API rejects, and because `--silent` exits 0 on failure the error is swallowed. Make the flag actually required with `.requiredOption(...)` (and reject omission in this test), or update the PR description/skill doc to say it is optional-on-the-CLI.</comment>

<file context>
@@ -0,0 +1,101 @@
+  });
+});
+
+it('still sends feedback without an objective', async () => {
+  const index = feedbackArgs.indexOf('--objective');
+  const withoutObjective = feedbackArgs.filter(
</file context>
Fix with cubic

const index = feedbackArgs.indexOf('--objective');
const withoutObjective = feedbackArgs.filter(
(_, i) => i !== index && i !== index + 1
);

expect((await cli(withoutObjective)).code).toBe(0);
expect(requests).toHaveLength(1);
expect(requests[0].body).not.toHaveProperty('objective');
});

it('rejects a blank objective before sending feedback', async () => {
const blank = [...feedbackArgs];
blank[feedbackArgs.indexOf('--objective') + 1] = ' ';

expect((await cli(blank)).code).not.toBe(0);
expect(requests).toHaveLength(0);
});
4 changes: 4 additions & 0 deletions src/commands/search-feedback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ export interface SearchFeedbackOptions {
valuableSources?: ValuableSourceInput[];
missingContent?: MissingContentInput[];
querySuggestions?: string;
objective?: string;
apiKey?: string;
apiUrl?: string;
output?: string;
Expand Down Expand Up @@ -129,6 +130,9 @@ export async function executeSearchFeedback(
if (options.querySuggestions) {
body.querySuggestions = options.querySuggestions;
}
if (options.objective) {
body.objective = options.objective;
}

const response = await fetch(url, {
method: 'POST',
Expand Down
15 changes: 14 additions & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
* Entry point for the CLI application
*/

import { Command, Option } from 'commander';
import { Command, InvalidArgumentError, Option } from 'commander';
import { createSqlCommand } from './commands/sql';
import { addFormatsAlias } from './utils/format-option';
import {
Expand Down Expand Up @@ -1401,6 +1401,13 @@ Examples:
return researchCmd;
}

function searchFeedbackObjective(value: string): string {

@cubic-dev-ai cubic-dev-ai Bot Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: searchFeedbackObjective duplicates the identical detail validator already defined in src/commands/alexandria-feedback.ts (same trim, same 2000-char check, same 'Use 1–2000 characters.' message and it's used for the same --objective purpose). Extract a shared text-validator helper into a utils module and reuse it in both places so the limit/message don't drift.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/index.ts, line 1404:

<comment>`searchFeedbackObjective` duplicates the identical `detail` validator already defined in src/commands/alexandria-feedback.ts (same trim, same 2000-char check, same 'Use 1–2000 characters.' message and it's used for the same `--objective` purpose). Extract a shared text-validator helper into a utils module and reuse it in both places so the limit/message don't drift.</comment>

<file context>
@@ -1401,6 +1401,13 @@ Examples:
   return researchCmd;
 }
 
+function searchFeedbackObjective(value: string): string {
+  const text = value.trim();
+  if (!text || text.length > 2000)
</file context>
Fix with cubic

const text = value.trim();
if (!text || text.length > 2000)

@cubic-dev-ai cubic-dev-ai Bot Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The >2000 rejection branch of searchFeedbackObjective has no test coverage. src/__tests__/search-feedback-objective.test.ts exercises only the missing and blank cases, so the newly added upper-bound check is untested.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/index.ts, line 1406:

<comment>The >2000 rejection branch of `searchFeedbackObjective` has no test coverage. `src/__tests__/search-feedback-objective.test.ts` exercises only the missing and blank cases, so the newly added upper-bound check is untested.</comment>

<file context>
@@ -1401,6 +1401,13 @@ Examples:
 
+function searchFeedbackObjective(value: string): string {
+  const text = value.trim();
+  if (!text || text.length > 2000)
+    throw new InvalidArgumentError('Use 1–2000 characters.');
+  return text;
</file context>
Fix with cubic

throw new InvalidArgumentError('Use 1–2000 characters.');
return text;
}

/**
* Create the search-feedback command. Used by agents (CLI, MCP, skills) to
* report search-result quality after a `firecrawl search` call. The first
Expand Down Expand Up @@ -1429,6 +1436,11 @@ function createSearchFeedbackCommand(): Command {
'--query-suggestions <text>',
'How the query or result set could be improved'
)
.option(
'--objective <text>',
'The underlying goal: what you or your user were ultimately trying to accomplish',
searchFeedbackObjective
)
.option(
'-k, --api-key <key>',
'Firecrawl API key (overrides global --api-key)'
Expand Down Expand Up @@ -1471,6 +1483,7 @@ function createSearchFeedbackCommand(): Command {
valuableSources,
missingContent,
querySuggestions: options.querySuggestions,
objective: options.objective,
apiKey: options.apiKey,
apiUrl: options.apiUrl,
output: options.output,
Expand Down
Loading