[righel]

Implements #506: isolation, bounded ingestion, and the operator controls and visibility (API and UI).

Why

A MISP feed fetch or server pull queued one task per event on the default queue. A few large sources could bury periodic jobs (indicator sync, warninglists, EDL refreshes, retention...) for hours, and nothing limited how many sources ingested at once.

What changes

Queues

• ingest: feed fetches, server pulls/pushes and their follow-ups (attribute/object handlers, correlation, reactor dispatches coming from ingestion). Served by a new ingest-worker service.
• maintenance: frequent periodic jobs (indicator and lookup cache sync, EDL refreshes, recent warninglist hits, the ingest watchdog...), served by the existing worker (--queues=celery,maintenance).
• bulk: rebuilds and reloads that walk everything and run for minutes to hours (correlations, indicator and lookup cache rebuilds, warninglist loads and full re-evaluations, galaxies, taxonomies, retention). Served by a new bulk-worker, so a rebuild can't hold up the frequent jobs.
• Each beat entry now expires after its own interval, so a backlog can't pile up repeated copies.

Ingestion runs (app/services/ingest_runs.py, Redis-backed)

• At most one run per source: fetching a feed that is already being fetched is skipped.
• At most ingest.max_concurrent_runs (runtime setting, default 2) runs ingest at once. The others wait as queued and retry every minute. ingest.paused holds all of them.
• A run's events go into a Redis list. Up to 2 drain tasks per run take them 10 at a time and requeue themselves, so the queue holds a few messages per run, not one per event, and concurrent runs take turns.
• Every 5 minutes a watchdog (revive_stalled_ingest_runs) restarts or closes runs with no progress for 45 minutes, for example after a drain task was killed mid-batch.
• Runs expire on their own after 3 hours without progress.

API

• GET /feeds/{id}/run and GET /servers/{id}/run return the current and last run: status, totals, pending count, error.
• DELETE on the same paths cancels a run. Disabling or deleting a feed or server also cancels its run.

UI

• Feeds and servers lists show each source's run in progress or how its last one ended. A feed's or server's page shows progress, a Cancel button, and the last run's outcome and error.
• Runtime settings has an ingest form: concurrent runs, pause.
• Diagnostics has a Queues card: messages waiting per queue, the workers consuming each (a queue with none is flagged), and the runs in progress or waiting (GET /diagnostics/queues).
• Cancelling closes a run at once. Each run has an id its drain tasks carry, so a leftover task from a cancelled run stops instead of draining a later run of the same source.

Feed event views

• The MISP feed explorer marks each event as ingested from this feed, newer in the feed, already here from another source, or not ingested, and counts how many were ingested. The feed row's magnifying glass now opens it, as on servers.
• A feed's page lists its 10 most recently ingested events (GET /feeds/{id}/events), with a link to all of them in Explore. ?range=all opens Explore without its default 30-day range.
• Events record created_at, when first written here; their timestamp is the source's own. It is in opensearch/mappings/misp-events.json, and existing indices pick it up through dynamic mapping.

Feed forms

• MISP feed rules are collapsed, with a summary in the header (e.g. Last 30 days).
• A separate, collapsed Advanced settings section holds authentication and indicator scoring, its header summarising what is set. The MISP server and organisation forms collapse their indicator scoring the same way.

Robustness

• Timeouts on feed downloads: 10 s to connect, 120 s to read. Remote MISP requests: 10 s / 300 s.
• A non-200 MISP feed manifest now fails the fetch with an error. Before, it was silently treated as empty.
• Objects a bulk ingest creates are handled in batches of 500 (handle_created_objects), like attributes. Before, one task per object (about 18,000 in 40 minutes for CIRCL OSINT) queued ahead of the drain tasks.
• reactor_dispatch rebuilds a cold trigger cache first. The gate fails open when the cache is missing (after a Redis restart, for example), and each ingested attribute then queued a dispatch.

Deployment

• docker-compose.yml adds ingest-worker (--queues=ingest, stop_grace_period: 5m so it can finish its batch on shutdown) and bulk-worker (--queues=bulk).
• Deployments with their own worker command need to serve maintenance, ingest and bulk too, for example a single worker with --queues=celery,maintenance,ingest,bulk; see the README.

Testing

• New tests/services/test_ingest_runs.py and reactor cache-warming tests, and a /diagnostics/queues test.
• Tests for GET /feeds/{id}/events and the explorer's local copies (they use the database, so CI runs them; not run locally).
• Each UI change was checked in a browser against the dev stack.
• Full suite in dev: 1711 passed. The 19 failures and errors occur the same way on main in the same stack: an oversized warninglist index on a 512 MB OpenSearch heap, and no modules service.
• End to end on the dev stack with CIRCL OSINT (131 events), Botvrij (12) and an unreachable feed, at a limit of 2 concurrent runs:
  • periodic jobs kept their schedule throughout;
  • the third run waited as queued;
  • a duplicate fetch was skipped;
  • the unreachable feed failed after the connect timeout;
  • stopping the worker let the current batch finish;
  • DELETE /feeds/{id}/run cancelled a run;
  • the watchdog closed a stalled run.

Closes #506

🤖 Generated with Claude Code

[github-actions]

OpenAPI changes

Added endpoints

• ALL /diagnostics/queues
• ALL /feeds/{feed_id}/events
• ALL /feeds/{feed_id}/run
• ALL /servers/{server_id}/run

Modified endpoints

• GET /events/
• POST /events/
• GET /events/{event_uuid}
• PATCH /events/{event_uuid}
• POST /servers/{server_id}/events/{event_uuid}/pull

[codecov]

Codecov Report

❌ Patch coverage is 93.23725% with 61 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.74%. Comparing base (f81cf71) to head (bc2dd05).

Files with missing lines	Patch %	Lines
api/app/worker/tasks.py	85.00%	30 Missing ⚠️
api/app/repositories/feeds.py	70.83%	14 Missing ⚠️
api/app/routers/diagnostics.py	84.37%	5 Missing ⚠️
api/app/routers/feeds.py	80.00%	4 Missing ⚠️
api/app/routers/servers.py	70.00%	3 Missing ⚠️
api/app/services/ingest_runs.py	98.20%	3 Missing ⚠️
api/app/repositories/reactor.py	71.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #507      +/-   ##
==========================================
+ Coverage   88.56%   88.74%   +0.18%     
==========================================
  Files         292      295       +3     
  Lines       34272    35114     +842     
==========================================
+ Hits        30354    31163     +809     
- Misses       3918     3951      +33     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.