Skip to content

fix(ci): stop the Dart suite racing the runtime's global panic hook - #413

Merged
Peefy merged 1 commit into
mainfrom
fix/dart-test-suite-serial
Oct 19, 2026
Merged

Peefy merged 1 commit into
mainfrom
fix/dart-test-suite-serial

Conversation

@Peefy

@Peefy Peefy commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Run the Dart test suite serially (dart/dart_test.yaml, concurrency: 1) — dart test runs test files in parallel isolates that share one libkcl, and overlapping evaluations clobber each other's panic hook, so a KCL error can come back with an empty message
  • Document the constraint in dart/README.md next to the existing not-thread-safe note
  • Drop the duplicated windows-latest matrix entry, which ran the Windows leg twice

The failure

Run #39233061134, Test on macos-15-intel:

❌ test/kcl_lib_test.dart: plugins reports an unknown method as a KCL error rather than crashing
Expected: contains 'nope'
  Actual: 'EvaluationError\n---> File ./__main__.k:2: \n\n'

thread '<unnamed>' panicked at crates/runtime/src/stdlib/plugin.rs:80:13:
invalid method: kcl_plugin.strings.nope is not found

Why

A plugin method that was never registered is reported the way every other KCL failure is — as a __kcl_PanicInfo__ object the plugin agent returns, which the runtime turns into a Rust panic. The panic's payload is what errMessage is built from, and the runtime captures it with a process-global std::panic hook that it takes, replaces and restores around every evaluation, recording into a thread-local.

Two overlapping evaluations therefore race on that hook: one thread restores the hook another is running under, and the panicking thread reads back an empty record, so the error renders with no message. Only the first panic on a thread exposes it — which is why the very next plugin test (reports a throwing method as a KCL error) passed.

Reproduced locally against libkcl v0.13.2 with four isolates (one driving plugin errors with a distinct method name per round, three hammering ordinary evaluations): 2 of 3 runs hit the race, one of them byte-identical to the CI failure at round 0.

Serial execution costs ~0.1s on this package.

Note

This stops the flake but is not the real fix: the hook install/restore is not concurrency-safe in kcl-lang/kcl (crates/runner/src/runner.rs), and FastRunner::run falls back to unwrap_or_default() instead of the caught panic payload when the record comes back empty. That belongs upstream and needs a new libkcl to take effect here.

Test plan

  • make test in dart/ — 141 tests pass
  • --reporter=expanded output shows each file's results as one contiguous block, i.e. suites really are serial now
  • matrix parses to 5 legs; windows-latest appears once

🤖 Generated with Claude Code

dart-test's macos-15-intel leg failed on "plugins reports an unknown method
as a KCL error rather than crashing": errMessage came back as an empty
EvaluationError while the real text ("invalid method: ... is not found") went
to stderr from the runtime's panic hook.

The plugin agent reports a missing method the way every other KCL failure is
reported — as a __kcl_PanicInfo__ object the runtime turns into a panic — and
that panic's payload is captured by a process-global std::panic hook that the
runtime installs and restores around every evaluation, recording into a
thread-local. `dart test` runs test files in parallel isolates sharing one
libkcl, so an overlapping evaluation restores the hook another thread is
running under; the panicking thread then reads an empty record and the KCL
error loses its message. Only the first panic on a thread shows it, which is
why the next plugin test passed.

Run the suite serially (~0.1s on this package) and document why in the README.
The upstream fix belongs in kcl-lang/kcl, where FastRunner::run should fall
back to the caught panic payload instead of unwrap_or_default().

Also drop the duplicated windows-latest matrix entry, which ran the Windows
leg twice.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Peefy
Peefy merged commit dab18d0 into main Oct 19, 2026
6 of 8 checks passed
@Peefy
Peefy deleted the fix/dart-test-suite-serial branch October 19, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant