Repository navigation
fix(indicators): trust changes reach every score, and aren't dropped - #489
Merged
Merged
Conversation
Changing a feed's, server's or organisation's score modifier or cap (or the scoring settings) queues a full rebuild. Two things kept the change from reaching the scores: - The rebuild only stored a score that moved by refresh_step (5) or more, the rule that keeps hourly decay from rewriting every indicator. A modifier of -3, or a cap of 60 on a value at 62, was computed and then dropped, while profiles, exports and lookups filter on the stored score. A requested rebuild now writes every score that moved; the hourly one keeps the step. - A request was merged into any rebuild already running, which had read the trust before the change, and the flag was only cleared when that run ended, so nothing else was queued. A queued rebuild that found a sync holding the lock gave up and left the flag set, suppressing further requests until the flag expired. Either way the change waited for the next hourly rebuild. A requested rebuild now clears the flag when it starts, so a later request queues the next run, and retries every 30 seconds while the lock is held. `app.cli rebuild-indicators` runs as a requested rebuild too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
OpenAPI changesNo changes to the OpenAPI spec. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #489 +/- ##
==========================================
+ Coverage 88.52% 88.56% +0.03%
==========================================
Files 291 292 +1
Lines 34172 34263 +91
==========================================
+ Hits 30252 30345 +93
+ Misses 3920 3918 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Saving a feed's, MISP server's or organisation's score modifier or cap, or the scoring settings, calls
request_rebuild(), which queues a full rebuild of the indicators. Two things kept the change from reaching the stored scores, which is what profiles, exports, sinks and/lookupfilter on.1. Small changes were computed and then dropped. A rebuild only stores a score that moved by
refresh_step(5) or more, or crossed the decayed threshold. That rule exists so hourly decay doesn't rewrite every indicator, but it also applied to the rebuild a trust change asked for. Two examples that never took effect until some later change moved the value by 5 or more:min_scoreof 61, although its only source is capped at 60.2. Requests could be lost.
REBUILD_QUEUED_KEYwas cleared only when a rebuild finished. There were two failure cases:rebuild()returnedNonewithout clearing the flag. The requested rebuild never ran, and later requests were suppressed until the flag expired (2 h).Either way the change only landed with the next hourly rebuild.
Fix
request_rebuild()queuesrebuild_indicators(requested=True)._Run(exact_scores=True)). The hourly run keepsrefresh_step, so incremental exports still carry only real changes.REBUILD_QUEUED_KEYwhen it starts, not when it ends. Requests stay merged until then, since the run reads the latest settings and trust at that moment; one arriving while it runs queues the next.REBUILD_RETRY_SECONDS) while a sync or another rebuild holds the lock, instead of giving up. The hourly run still just skips a turn.app.cli rebuild-indicatorsruns as a requested rebuild.docs/features/indicators.mddescribes when scores are rewritten after a trust change.Testing
app/tests/repositories/test_indicator_rebuilds.py: mock-based, 7 tests, passed locally. It covers:test_indicator_scoring.py: the tests now patchapply_asyncand checkrequested=True.test_organisation_trustalso checks end to end that a 3-point trust change (53 → 56) is skipped by the hourly rebuild and written by a requested one. This is anApiTestertest, so it wasn't run locally (the suite wipes the dev data); CI runs it.🤖 Generated with Claude Code