feat(cli): submit keyless job feedback by Max17190 · Pull Request #221 · firecrawl/cli

Max17190 · 2026-09-11T20:14:45Z

Why

Let agents discover and optionally submit evidence for keyless Search, Scrape, and Parse using the API contract in #4616.

Summary

Extend the existing firecrawl feedback command with task, assessment, observation JSON/files, and Parse document class. Keyless submissions omit Authorization.
Preserve job references and invitations in JSON output and stderr, including failed jobs, while preserving normal content on stdout.
Align optional feedback guidance with the API in command help, reminders, README, and bundled skills. Reuse one short help string.
Default keyless Search to web results while retaining authenticated defaults, explicit sources, and the Map feedback auth gate.
Preserve API validation details and retry timing for keyless rejections. Preserve authenticated feedback contracts and preferences. Keep the preference helper in the existing command module.

Release after the API is deployed and Docs #1414 is published.

Test Plan

Current head 713628d: synced with main; all 704 tests across 38 files passed after rebuilding, plus build, TypeScript checks, formatting, and diff whitespace checks.
Coverage includes keyless requests, same-request references on success/failure, evidence flags and help, JSON output, API validation/retry details, case-normalized auth gates, authenticated Search defaults, and feedback preferences.
All 861 API/OpenAPI schema parity cases passed across the aligned API and Docs branches.
Clarified submission requirements, the invitation deadline, and the same-caller-IP requirement. Tests restore the original feedback preference environment variable.
Cubic reports zero new issues on the final revision; all seven CI jobs and the Cubic check passed.
Built-client exercises against the staging API passed for Search, Scrape, Parse, failed-job references, accepted feedback, and duplicate retries.

After an eligible call, print the job reference and a one-line optional reminder pointing to `firecrawl feedback <endpoint> <jobId> --help` instead of the full server message and flag template. Remove the daily submission rule and the exact attempt rate from help, the README, and bundled skills, and drop the retired DAILY_LIMIT_REACHED code.

Resolve conflicts with Alexandria session feedback and the Alexandria-only search command: - Send Alexandria session fields for endpoint alexandria and keyless task, assessment, document class, and observations for job feedback. - Keep the keyless feedback help beside the updated search action. - Keep the Alexandria guidance and relocated completion criteria in the bundled skills alongside the keyless feedback contract. - Describe keyless feedback as requested in exchange for free keyless use in the stderr reminder, help, README, and skills.

Point to feedback when a result is wrong, incomplete, blocked, or an error, matching the API invitation and MCP wording, instead of framing keyless use as an exchange for feedback. Updates the reminder, command help, README and skills.

cubic-dev-ai

Review completed against the latest diff

_{Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 18 files

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked because this review re-detected 1 unresolved P0–P2 issue already reported by Cubic.

Re-trigger cubic}

Max17190 · 2026-10-01T16:50:00Z

@cubic-dev-ai review this PR in full at the current head.

Use the API contract in firecrawl/firecrawl#4616 as the source of truth. Keyless feedback is optional for Search, Scrape, and Parse. Map feedback remains authenticated. Keyless evidence requirements are checked before submission; authenticated fields and preferences are retained. Search JSON preserves references even for empty results. Successful Scrape invitations are in data.metadata; failed jobs carry top-level metadata and are handled by the shared keyless request path, including HTTP 200 responses with success: false.

Please recheck the complete diff and the prior findings against the current implementation.

cubic-dev-ai · 2026-10-01T16:50:11Z

@cubic-dev-ai review this PR in full at the current head.

Use the API contract in firecrawl/firecrawl#4616 as the source of truth. Keyless feedback is optional for Search, Scrape, and Parse. Map feedback remains authenticated. Keyless evidence requirements are checked before submission; authenticated fields and preferences are retained. Search JSON preserves references even for empty results. Successful Scrape invitations are in data.metadata; failed jobs carry top-level metadata and are handled by the shared keyless request path, including HTTP 200 responses with success: false.

Please recheck the complete diff and the prior findings against the current implementation.
...

@Max17190 I have started the AI code review. It will take a few minutes to complete.

Max17190 · 2026-10-01T16:58:05Z

@cubic-dev-ai review this PR in full at the current head.

Use the API contract in firecrawl/firecrawl#4616 as the source of truth. Keyless feedback is optional for Search, Scrape, and Parse. Map feedback remains authenticated. Keyless evidence requirements are checked before submission; authenticated fields and preferences are retained. Search JSON preserves references even for empty results. Successful Scrape invitations are in data.metadata; failed jobs carry top-level metadata and are handled by the shared keyless request path, including HTTP 200 responses with success: false.

The latest revision also displays keyless validation details and retry timing while preserving authenticated error output.

Please recheck the complete diff and the prior findings against the current implementation.

cubic-dev-ai · 2026-10-01T16:58:15Z

@cubic-dev-ai review this PR in full at the current head.

Use the API contract in firecrawl/firecrawl#4616 as the source of truth. Keyless feedback is optional for Search, Scrape, and Parse. Map feedback remains authenticated. Keyless evidence requirements are checked before submission; authenticated fields and preferences are retained. Search JSON preserves references even for empty results. Successful Scrape invitations are in data.metadata; failed jobs carry top-level metadata and are handled by the shared keyless request path, including HTTP 200 responses with success: false.

The latest revision also displays keyless validation details and retry timing while preserving authenticated error output.
...

@Max17190 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 19 files

Confidence score: 5/5

In parse.test.ts, the keyless parse path never reads FIRECRAWL_NO_ENDPOINT_FEEDBACK or sends an opt-out header, so this test doesn't verify the env var's effect. Update it to assert behavior the keyless path actually supports.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/__tests__/commands/parse.test.ts">

<violation number="1" location="src/__tests__/commands/parse.test.ts:77">
P3: This test stubs `FIRECRAWL_NO_ENDPOINT_FEEDBACK` and asserts the request headers, but the keyless parse path never reads that env var and never sends an opt-out header, so the stub has no effect and the assertion is an exact duplicate of the "posts to /v2/parse without auth" test's header check. Either remove the test or make it assert the actual opt-out-relevant behavior of this PR: the keyless parse still reports the feedback invitation on stderr even when the opt-out env is set (as `feedback-invitation.test.ts` pins down).</violation>
</file>

_{Shadow auto-approve: would require human review. Adds keyless feedback submissions with new required fields and an auth bypass for feedback on search/scrape/parse; needs human review of the authorization and API contract changes.

Fix all with cubic | Re-trigger cubic}

… tests

Max17190 · 2026-10-01T17:15:13Z

@cubic-dev-ai review the latest changes at the current head.

Removed the redundant Parse opt-out test reported in the latest review and the equivalent shared-request header tests. Invitation behavior remains covered by the helper and command tests plus actual API staging flows. The feedback auth gate now uses the same case normalization as the existing endpoint parser; regression tests verify that Search, Scrape, and Parse case variants remain keyless while Map variants retain authentication.

cubic-dev-ai · 2026-10-01T17:15:27Z

@cubic-dev-ai review the latest changes at the current head.

Removed the redundant Parse opt-out test reported in the latest review and the equivalent shared-request header tests. Invitation behavior remains covered by the helper and command tests plus actual API staging flows. The feedback auth gate now uses the same case normalization as the existing endpoint parser; regression tests verify that Search, Scrape, and Parse case variants remain keyless while Map variants retain authentication.

@Max17190 A review is already in progress. Try again after it finishes.

cubic-dev-ai

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds keyless feedback submission for search/scrape/parse, a new required-field contract, and an auth-gate bypass for feedback; this is an authorization/data-handling change tied to an unreleased API contract.

Re-trigger cubic}

…edback

Max17190 · 2026-10-03T03:01:59Z

@cubic-dev-ai review the complete current PR diff after syncing with main. Use firecrawl/firecrawl#4616 as the source of truth for optional keyless feedback. Verify the 24-hour window, evidence requirements, preserved job references on success and failure, ownership, one submission per job, separate attempt limits, no keyless refunds or quota resets, and preservation of authenticated feedback. Recheck prior findings against the current code.

cubic-dev-ai · 2026-10-03T03:02:08Z

@cubic-dev-ai review the complete current PR diff after syncing with main. Use firecrawl/firecrawl#4616 as the source of truth for optional keyless feedback. Verify the 24-hour window, evidence requirements, preserved job references on success and failure, ownership, one submission per job, separate attempt limits, no keyless refunds or quota resets, and preservation of authenticated feedback. Recheck prior findings against the current code.

@Max17190 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 18 files

Confidence score: 4/5

The README.md can be read as starting the 24-hour window when the reference arrives, but the API measures it from job creation. Clarify that submissions are due within 24 hours of job creation.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="README.md">

<violation number="1" location="README.md:476">
P2: This can imply a fresh 24-hour window after receiving the reference, but the API deadline is based on the job timestamp. State that submissions must be made within 24 hours of job creation.</violation>
</file>

_{Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic}

cubic-dev-ai

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved P0–P2 issue from previous reviews.

Re-trigger cubic}

feat(cli): submit keyless job feedback

cf5a9ac

Max17190 mentioned this pull request Sep 11, 2026

feat(api): accept structured keyless feedback firecrawl/firecrawl#4616

Open

3 tasks

Max17190 added 16 commits September 11, 2026 15:23

docs(cli): distinguish keyless and authenticated feedback

521389c

fix(cli): forward feedback invitation preferences

cf929f4

docs(cli): clarify shared keyless feedback daily limit

02baf67

fix(cli): preserve keyless feedback and silence opted-out guidance

2f3855c

fix(cli): retain keyless invitations and optional submissions

aec2df9

feat(cli): support structured keyless feedback categories

80a8808

docs(cli): clarify search feedback source attribution

fc5f679

docs(cli): align keyless feedback evidence guidance

b9a61bb

docs: clarify keyless feedback limits and failure reasons

49c4737

merge: synchronize keyless feedback with main

328a8b8

style(test): format CLI terms refusal test

7a0847d

merge: sync main for keyless feedback

9ea44ef

refactor(feedback): simplify and align optional keyless guidance

8f6ce95

Max17190 marked this pull request as ready for review October 1, 2026 16:08

fix(search): use web results by default for keyless callers

b0d54d5

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

fix(feedback): retain auth boundaries and handle keyless failures

4cb90b3

fix(feedback): expose keyless validation details and retry timing

66a980f

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread src/__tests__/commands/parse.test.ts Outdated

fix(feedback): preserve keyless endpoint aliases and remove redundant…

ba84855

… tests

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into max/enable-keyless-fe…

15c817d

…edback

cubic-dev-ai Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread skills/firecrawl-search/SKILL.md Outdated

Comment thread README.md Outdated

Comment thread README.md

Comment thread src/__tests__/utils/feedback-invitation.test.ts Outdated

fix: clarify keyless feedback submission requirements

713628d

cubic-dev-ai Bot reviewed Oct 3, 2026

View reviewed changes

Conversation

Max17190 commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Summary

Test Plan

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Max17190 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

Max17190 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Max17190 commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 1, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Max17190 commented Oct 3, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 3, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Max17190 commented Sep 11, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading