fix(ci): make memory-gate visible, fail loudly, stop starving roles_calculation - #170
Open
Oddly wants to merge 1 commit into
Open
fix(ci): make memory-gate visible, fail loudly, stop starving roles_calculation#170Oddly wants to merge 1 commit into
Oddly wants to merge 1 commit into
Conversation
…es_calculation The molecule capacity-gate retry loop in shared/create.yml was set to retries: 360 (3 hours) but the workflows that call it have timeouts of 20-120 minutes, so a starved scenario was always killed by the workflow timeout before its own retries ran out. On top of that, Ansible's FAILED - RETRYING messages sit in Python's stdout buffer under the non-TTY GHA log, so the run looked like a silent hang for 45+ minutes before the last cancel showed up. Between them, a queue-starvation event and a genuinely broken scenario were impossible to tell apart. Cap retries at 30 (= 15 min) so the task fails loudly with the last 'No capacity: X committed + Y needed > Z available' stdout, well inside every workflow's timeout. Set PYTHONUNBUFFERED=1 in the molecule workflow env so the retry chatter surfaces in real time. Also shrink the four elasticsearch_roles_calculation containers from 4 GB to 2 GB each (16 GB -> 8 GB). elasticsearch_heap is 1 GB and the scenario only exercises the role-calculation code path, so 2 GB is plenty of headroom. Halving the memory ask is what lets the scenario clear the gate under peak concurrency instead of retrying until the workflow timeout.
Oddly
enabled auto-merge (squash)
August 4, 2026 07:12
📝 WalkthroughWalkthroughMolecule CI now uses unbuffered Python output, lower memory allocations for the Elasticsearch calculation scenario, and a 15-minute container launch retry window. ChangesMolecule CI tuning
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@molecule/elasticsearch_roles_calculation/molecule.yml`:
- Around line 19-37: Update the elasticsearch_roles_calculation reservation in
scripts/wait-for-memory.sh to 8192 MB, matching the four 2048 MB Molecule
platforms defined in molecule.yml; preferably derive the value from the Molecule
configuration to prevent future drift.
In `@molecule/shared/create.yml`:
- Around line 129-138: The retry configuration around _launch_result must
enforce a true elapsed-time limit rather than relying only on retries and delay.
Update the launch command’s flock wait budget or add a deadline so lock
contention cannot extend the task toward the workflow timeout, while preserving
the existing retry behavior and ensuring the total standard-workflow duration
remains within the intended 15-minute guard.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 15bb8583-1ed8-4db7-bbef-6dab38c5c6b1
📒 Files selected for processing (3)
.github/workflows/molecule.ymlmolecule/elasticsearch_roles_calculation/molecule.ymlmolecule/shared/create.yml
Comment on lines
+19
to
+37
| memory_mb: 2048 | ||
| - name: "es-calc2-${MOLECULE_DISTRO:-debian12}-r${ELASTIC_RELEASE:-9}${MOLECULE_RUN_SUFFIX}" | ||
| groups: | ||
| - elasticsearch | ||
| distro: "${MOLECULE_DISTRO:-debian12}" | ||
| memory_mb: 4096 | ||
| memory_mb: 2048 | ||
| - name: "es-calc3-${MOLECULE_DISTRO:-debian12}-r${ELASTIC_RELEASE:-9}${MOLECULE_RUN_SUFFIX}" | ||
| groups: | ||
| - elasticsearch | ||
| distro: "${MOLECULE_DISTRO:-debian12}" | ||
| memory_mb: 4096 | ||
| memory_mb: 2048 | ||
| # Separate single-node "monitoring" cluster, run in a second play against | ||
| # this group to catch the regression from #143 where group_by accumulated | ||
| # elasticsearch_role_master across plays. | ||
| - name: "es-calc-mon-${MOLECULE_DISTRO:-debian12}-r${ELASTIC_RELEASE:-9}${MOLECULE_RUN_SUFFIX}" | ||
| groups: | ||
| - elasticsearch_mon | ||
| distro: "${MOLECULE_DISTRO:-debian12}" | ||
| memory_mb: 4096 | ||
| memory_mb: 2048 |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
Synchronize the capacity reservation with the new total.
These four platforms now request 4 * 2048 = 8192 MB. However, scripts/wait-for-memory.sh:43-59 still reserves 16384 MB for elasticsearch_roles_calculation. The workflow uses that reservation before molecule converge, so the scenario can remain queued as a 16 GB job and the intended capacity-gate improvement is not achieved.
Update the reservation to 8192 MB, or derive it from the Molecule configuration to prevent drift.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@molecule/elasticsearch_roles_calculation/molecule.yml` around lines 19 - 37,
Update the elasticsearch_roles_calculation reservation in
scripts/wait-for-memory.sh to 8192 MB, matching the four 2048 MB Molecule
platforms defined in molecule.yml; preferably derive the value from the Molecule
configuration to prevent future drift.
Comment on lines
+129
to
138
| # 30 retries × 30s = 15 min. Deliberately shorter than the workflow | ||
| # timeout (45 min for the standard molecule.yml call, 120 min for | ||
| # full_stack) so the task fails loudly with the last "No capacity: | ||
| # …" stdout instead of being silently cancelled by the workflow | ||
| # timeout — that made the queue-starvation cases very hard to | ||
| # diagnose (see PR investigating the elasticsearch_roles_calculation | ||
| # / elasticstack_default post-outage hangs). | ||
| retries: 30 | ||
| delay: 30 | ||
| until: _launch_result.rc == 0 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files matching molecule/shared/create.yml:"
git ls-files | grep -Fx 'molecule/shared/create.yml' || true
echo
echo "Context around create.yml lines 60-150:"
if [ -f molecule/shared/create.yml ]; then
nl -ba molecule/shared/create.yml | sed -n '60,160p'
fi
echo
echo "Search workflow timeout mentions:"
rg -n "45 min|120 min|timeout|full_stack|molecule.yml|retry|delay|flock -x -w 60|_launch_result" molecule .github 2>/dev/null || trueRepository: Oddly/elasticstack
Length of output: 308
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Context around molecule/shared/create.yml lines 60-150:"
awk '{printf "%6d\t%s\n", NR, $0}' molecule/shared/create.yml | sed -n '60,150p'
echo
echo "Context around launch/flock tasks and retries/delay:"
awk '{printf "%6d\t%s\n", NR, $0}' molecule/shared/create.yml | sed -n '1,180p'
echo
echo "Search workflow timeout mentions:"
rg -n "45 min|120 min|timeout|full_stack|molecule.yml|retry|delay|flock -x -w 60|_launch_result|molecule/shared/create.yml" molecule .github 2>/dev/null || trueRepository: Oddly/elasticstack
Length of output: 27420
Bound the retry window by elapsed time.
retries: 30 and delay: 30 count attempts and inter-attempt waits, but the command can also wait up to 60 seconds at flock -x -w 60 each time. Under lock contention, this task can run close to 45 minutes before Ansible gives up, which defeats the 15-minute guard for the standard workflow. Use an elapsed-time deadline, or reduce the lock/wait budget so the total time stays below the workflow timeout.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@molecule/shared/create.yml` around lines 129 - 138, The retry configuration
around _launch_result must enforce a true elapsed-time limit rather than relying
only on retries and delay. Update the launch command’s flock wait budget or add
a deadline so lock contention cannot extend the task toward the workflow
timeout, while preserving the existing retry behavior and ensuring the total
standard-workflow duration remains within the intended 15-minute guard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While chasing the post-outage "elasticsearch_roles_calculation" cancellations on #155, I ended up finding that the memory-based capacity gate was doing three things wrong in combination.
The retry loop in
molecule/shared/create.ymlis set toretries: 360, delay: 30, i.e. three hours. Every workflow that callsmolecule.ymlhas a timeout in the 20-120 minute range, so a scenario that failed the memory check would silently retry until the workflow itself was killed. The user got a "The operation was canceled" message with zero context about what actually went wrong. Cuttingretriesto 30 means the task fails loudly after 15 minutes with the last "No capacity: X committed + Y needed > Z available" stdout, well inside the shortest workflow timeout.Second, Ansible runs unbuffered through Python and GitHub Actions logs are not a TTY, so the
FAILED - RETRYING (N retries left)messages sit in the stdout buffer and only flush when the buffer fills. On the two hangs I looked at, 73 retry lines all appeared at the same microsecond timestamp after roughly an hour of "silence."PYTHONUNBUFFERED=1in the workflowenv:fixes this so the retry chatter surfaces in real time.Third,
elasticsearch_roles_calculationasks for4 × 4 GB = 16 GBeven though its heap is 1 GB and the scenario only exercises the role-count calculation. Under peak concurrency the 16 GB request never won a slot against the 6-8 GB scenarios that kept flowing in and out. Halving each container to 2 GB drops the ask to 8 GB total, which clears the gate under the same load.Together the three changes should turn "job cancelled after 45 minutes of silence" into either "runs green" or "fails loudly with a real error" — no more guessing which of the two is happening.
Summary by CodeRabbit