Skip to content

Improve integration test reliability and diagnostics - #269

Merged
David Negstad (danegsta) merged 3 commits into
mainfrom
danegsta-actions-failure-audit
Sep 22, 2026
Merged

David Negstad (danegsta) merged 3 commits into
mainfrom
danegsta-actions-failure-audit

Conversation

@danegsta

Copy link
Copy Markdown
Member

Recent CI failures exposed several reliability gaps: standard integration environments could still be shutting down after a test completed, the physical-volume deletion test could observe an intermediate controller state, and intermittent PTY failures did not report enough process information to identify the failing layer.

This change:

  • makes standard test environments own cancellation and synchronously wait for API server disposal during test cleanup, with an explicit post-shutdown callback phase for teardown assertions
  • waits for the complete expected terminating/creating volume state instead of asserting against the first observed deletion timestamp
  • adds failure-only PTY diagnostics covering read attempts, bytes received, elapsed time, child PID, exit status, exit error, and captured output

The historical PTY empty-EOF failure was not reproduced locally. Sustained concurrent stress did expose a separate /dev/ptmx allocation-pressure failure, so the new diagnostics are intended to distinguish child execution, PTY delivery, and process-exit behavior the next time CI encounters the original failure.

Validation:

  • make lint
  • make test
  • volume deletion test passed in ten separate test processes

Centralize standard test environment shutdown, wait for the complete volume deletion state, and add PTY failure diagnostics for future CI investigation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The behavioral changes are sound; the identified cleanup duplication and error-message casing are non-blocking refinements.

Review effort: Balanced
Findings: 1 Medium severity · 1 Low severity

Open (2)
What changed in this PR

Improves integration-test teardown reliability, volume-state synchronization, and PTY failure diagnostics.

Changes:

  • Centralizes synchronous test-environment shutdown and post-shutdown assertions.
  • Waits for the complete expected volume deletion state.
  • Adds detailed PTY read and child-process diagnostics.
File Description
test/​integration/​standard_test_env.go Adds cleanup-owned environment shutdown and callbacks.
test/​integration/​controllers_common_test.go Preserves manual lifecycle for TestMain.
test/​integration/​container_network_tunnel_proxy_test.go Migrates teardown and post-shutdown assertions.
test/​integration/​container_controller_test.go Uses cleanup-aware environment startup.
test/​integration/​executable_controller_test.go Uses cleanup-aware environment startup.
test/​integration/​network_controller_test.go Uses cleanup-aware environment startup.
test/​integration/​volume_controller_test.go Uses cleanup-aware environment startup.
test/​integration/​v2_physical_container_controller_test.go Migrates dedicated environment cleanup.
test/​integration/​v2_physical_container_image_controller_test.go Migrates dedicated environment cleanup.
test/​integration/​v2_physical_container_network_controller_test.go Uses cleanup-aware environment startup.
test/​integration/​v2_physical_container_volume_controller_test.go Strengthens deletion-state synchronization.
internal/​termpty/​pty_test.go Adds reusable PTY failure diagnostics.
internal/​termpty/​pty_unix_test.go Applies diagnostics to Unix output tests.
internal/​termpty/​conpty_windows_test.go Applies diagnostics to Windows output tests.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/integration/standard_test_env.go
Comment thread internal/termpty/pty_test.go Outdated
Rely on centralized standard environment cleanup and follow lowercase error string conventions for PTY diagnostics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Context-timeout failures still bypass collection of the child process exit status and error.

Review effort: Balanced
Findings: None

Resolved since last review (2)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Wait for process exit after context expiration read failure

internal/​termpty/​pty_test.go:194

When readUntil fails because ctx expired, this same ctx.Done() case is already ready, so the diagnostic path returns immediately without waiting for the context-driven process termination. That failure mode therefore omits the child exit status/error the helper is intended to capture. Keep the bounded exitTimer, but wait for either Exited() or that timer after a read failure.

After a read timeout, keep waiting for either the child exit notification or the existing bounded diagnostic timer instead of immediately selecting the expired context.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The cleanup migration is consistent across callers, and no unresolved correctness issues were identified.

Review effort: Balanced
Findings: None

@danegsta
David Negstad (danegsta) merged commit e7193a8 into main Sep 22, 2026
13 checks passed
@danegsta
David Negstad (danegsta) deleted the danegsta-actions-failure-audit branch September 22, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants