Skip to content

fix(ci): make memory-gate visible, fail loudly, stop starving roles_calculation #170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Oddly wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix(ci): make memory-gate visible, fail loudly, stop starving roles_calculation #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7d044ad
fix(ci): make memory-gate visible, fail loudly, and stop starving rol…
Oddly Aug 4, 2026
2fa93a3
fix(ci): raise test_full_stack timeout to 180 min
Oddly Aug 4, 2026
6f23219
fix(ci): cap concurrent full_stack matrix jobs at 6
Oddly Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/molecule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,12 @@ jobs:
env:
COLLECTION_NAMESPACE: oddly
COLLECTION_NAME: elasticstack
# Ansible/molecule are Python; without PYTHONUNBUFFERED the
# "FAILED - RETRYING (N retries left)" messages that Ansible emits
# from the wait-for-capacity retry loop sit in Python's stdout
# buffer and only flush when the buffer fills, making the log look
# like a silent hang for tens of minutes.
PYTHONUNBUFFERED: '1'
ANSIBLE_PIPELINING: 'true'
ANSIBLE_GATHERING: smart
ANSIBLE_ANY_ERRORS_FATAL: 'true'
Expand Down
15 changes: 14 additions & 1 deletion .github/workflows/test_full_stack.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,13 @@ jobs:
runs-on: self-hosted
needs: [changes, lint_full]
if: needs.changes.outputs.should_test == 'true' || github.event_name != 'pull_request'
timeout-minutes: 120
# 180 min: elasticstack_default is the heaviest scenario and observed
# runs range 70-105 min in isolation, but the extra queue wait added
# by the memory-capacity gate on a busy runner can push a single job
# past the previous 120-min cap. The memory-gate retry cap in
# shared/create.yml is now 15 min so this ceiling is only reached
# when the scenario genuinely runs long, not when the queue starves it.
timeout-minutes: 180
Comment on lines +66 to +72

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Use the reusable Molecule workflow for this timeout.

This job runs Molecule directly and sets timeout-minutes locally. .github/workflows/molecule.yml already exposes a timeout input and applies it to the reusable job. Pass 180 through that workflow instead of maintaining a second Molecule implementation. This keeps runner, environment, cleanup, matrix, and timeout behavior under one workflow contract.

As per path instructions, Molecule test workflows should use the reusable molecule.yml workflow.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/test_full_stack.yml around lines 66 - 72, Update the
workflow invoking Molecule directly to call the reusable molecule.yml workflow
instead, passing 180 through its exposed timeout input. Remove the local
timeout-minutes setting and preserve the existing scenario, runner, environment,
cleanup, and matrix configuration through the reusable workflow contract.

Source: Path instructions


env:
COLLECTION_NAMESPACE: oddly
Expand All @@ -85,6 +91,13 @@ jobs:

strategy:
fail-fast: false
# Each scenario/distro/release combo needs 15-25 GB of container RAM on
# the shared incus-ci host. The memory-capacity gate in shared/create.yml
# eventually admits every job, but without a matrix-level cap the full
# 16-combo PR matrix would swamp the 131 GB host and starve the biggest
# scenarios (elasticstack_default, cert_renewal). 6 concurrent slots keep
# committed memory around 90-100 GB with head-room to spare.
max-parallel: 6
matrix:
distro: ${{ (github.event_name == 'pull_request' || github.event_name == 'merge_group') && fromJSON('["rockylinux9","debian13"]') || (inputs.distros != '' && fromJSON(inputs.distros)) || fromJSON('["rockylinux9","ubuntu2204","ubuntu2404","ubuntu2604","debian12","debian13"]') }}
scenario:
Expand Down
13 changes: 9 additions & 4 deletions molecule/elasticsearch_roles_calculation/molecule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,29 +7,34 @@ dependency:
driver:
name: default
platforms:
# 2 GB per container: the scenario runs elasticsearch_heap: 1 (= 1 GB)
# and only exercises the role-count-calculation code path, not any
# ingest workload. The default 4 GB was making the 16 GB total block
# the memory-capacity gate under peak concurrency and starve the
# scenario until the 45-min workflow timeout cancelled it.
- name: "es-calc1-${MOLECULE_DISTRO:-debian12}-r${ELASTIC_RELEASE:-9}${MOLECULE_RUN_SUFFIX}"
groups:
- elasticsearch
distro: "${MOLECULE_DISTRO:-debian12}"
memory_mb: 4096
memory_mb: 2048
- name: "es-calc2-${MOLECULE_DISTRO:-debian12}-r${ELASTIC_RELEASE:-9}${MOLECULE_RUN_SUFFIX}"
groups:
- elasticsearch
distro: "${MOLECULE_DISTRO:-debian12}"
memory_mb: 4096
memory_mb: 2048
- name: "es-calc3-${MOLECULE_DISTRO:-debian12}-r${ELASTIC_RELEASE:-9}${MOLECULE_RUN_SUFFIX}"
groups:
- elasticsearch
distro: "${MOLECULE_DISTRO:-debian12}"
memory_mb: 4096
memory_mb: 2048
# Separate single-node "monitoring" cluster, run in a second play against
# this group to catch the regression from #143 where group_by accumulated
# elasticsearch_role_master across plays.
- name: "es-calc-mon-${MOLECULE_DISTRO:-debian12}-r${ELASTIC_RELEASE:-9}${MOLECULE_RUN_SUFFIX}"
groups:
- elasticsearch_mon
distro: "${MOLECULE_DISTRO:-debian12}"
memory_mb: 4096
memory_mb: 2048
Comment on lines +19 to +37

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

Synchronize the capacity reservation with the new total.

These four platforms now request 4 * 2048 = 8192 MB. However, scripts/wait-for-memory.sh:43-59 still reserves 16384 MB for elasticsearch_roles_calculation. The workflow uses that reservation before molecule converge, so the scenario can remain queued as a 16 GB job and the intended capacity-gate improvement is not achieved.

Update the reservation to 8192 MB, or derive it from the Molecule configuration to prevent drift.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@molecule/elasticsearch_roles_calculation/molecule.yml` around lines 19 - 37,
Update the elasticsearch_roles_calculation reservation in
scripts/wait-for-memory.sh to 8192 MB, matching the four 2048 MB Molecule
platforms defined in molecule.yml; preferably derive the value from the Molecule
configuration to prevent future drift.

provisioner:
name: ansible
env:
Expand Down
9 changes: 8 additions & 1 deletion molecule/shared/create.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,14 @@
REMOTE_SCRIPT
changed_when: true
register: _launch_result
retries: 360
# 30 retries × 30s = 15 min. Deliberately shorter than the workflow
# timeout (45 min for the standard molecule.yml call, 120 min for
# full_stack) so the task fails loudly with the last "No capacity:
# …" stdout instead of being silently cancelled by the workflow
# timeout — that made the queue-starvation cases very hard to
# diagnose (see PR investigating the elasticsearch_roles_calculation
# / elasticstack_default post-outage hangs).
retries: 30
delay: 30
until: _launch_result.rc == 0
Comment on lines +129 to 138

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files matching molecule/shared/create.yml:"
git ls-files | grep -Fx 'molecule/shared/create.yml' || true

echo
echo "Context around create.yml lines 60-150:"
if [ -f molecule/shared/create.yml ]; then
  nl -ba molecule/shared/create.yml | sed -n '60,160p'
fi

echo
echo "Search workflow timeout mentions:"
rg -n "45 min|120 min|timeout|full_stack|molecule.yml|retry|delay|flock -x -w 60|_launch_result" molecule .github 2>/dev/null || true

Repository: Oddly/elasticstack

Length of output: 308

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Context around molecule/shared/create.yml lines 60-150:"
awk '{printf "%6d\t%s\n", NR, $0}' molecule/shared/create.yml | sed -n '60,150p'

echo
echo "Context around launch/flock tasks and retries/delay:"
awk '{printf "%6d\t%s\n", NR, $0}' molecule/shared/create.yml | sed -n '1,180p'

echo
echo "Search workflow timeout mentions:"
rg -n "45 min|120 min|timeout|full_stack|molecule.yml|retry|delay|flock -x -w 60|_launch_result|molecule/shared/create.yml" molecule .github 2>/dev/null || true

Repository: Oddly/elasticstack

Length of output: 27420

Bound the retry window by elapsed time.

retries: 30 and delay: 30 count attempts and inter-attempt waits, but the command can also wait up to 60 seconds at flock -x -w 60 each time. Under lock contention, this task can run close to 45 minutes before Ansible gives up, which defeats the 15-minute guard for the standard workflow. Use an elapsed-time deadline, or reduce the lock/wait budget so the total time stays below the workflow timeout.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@molecule/shared/create.yml` around lines 129 - 138, The retry configuration
around _launch_result must enforce a true elapsed-time limit rather than relying
only on retries and delay. Update the launch command’s flock wait budget or add
a deadline so lock contention cannot extend the task toward the workflow
timeout, while preserving the existing retry behavior and ensuring the total
standard-workflow duration remains within the intended 15-minute guard.


Expand Down
Loading