Skip to content

fix: enforce --timeout as an idle limit without busy-waiting - #185

Open
hohieuai wants to merge 1 commit into
googlecolab:mainfrom
hohieuai:fix/timeout-idle-limit
Open

hohieuai wants to merge 1 commit into
googlecolab:mainfrom
hohieuai:fix/timeout-idle-limit

Conversation

@hohieuai

@hohieuai hohieuai commented Oct 24, 2026 •

Copy link
Copy Markdown

Fixes #183.

Summary

--timeout for colab run and colab exec is documented as an idle threshold: #38 introduced it so users can "increase the idle execution threshold" for silent tasks, and the design docs describe it as protecting long-running silent tasks from being "prematurely killed". Runtime.execute_code, however, passes the value to jupyter-kernel-client's execute_interactive, which treats it as a deadline for the whole execution.

This PR enforces --timeout as the idle limit that the docs describe, and removes the busy wait that the pinned library enters once its deadline passes. The flag and its 30-second default are unchanged.

Problem

As measured in #183:

  • With 0.8.0 (Google's fork, used for development) and 0.9.0 (the PyPI pin): the deadline is never enforced. Once it passes, the client polls with a zero timeout and holds a CPU core at about 100% until the execution ends.
  • With 1.0.2, which fixes that loop (Fix interactive execution timeout spin datalayer/jupyter-kernel-client#41): the deadline is enforced as a limit on the whole run. A run that prints every 5 seconds fails after 30 seconds.

Changes

  • src/colab_cli/runtime.py: for streaming execution without stdin (colab run, colab exec, colab repl), execute_code calls a new _execute_streaming method. It works on the websocket client behind the kernel client, as _apply_ws_hook already does, through execute, get_iopub_msg and get_shell_msg. These methods are the same in 0.8.0, 0.9.0 and 1.0.2.
    • It blocks for at most one second per call, so waiting uses no CPU.
    • Every message from the execution resets the idle clock. Messages from other requests are ignored.
    • With no message for timeout seconds, it raises TimeoutError. The message keeps its Timeout waiting for output prefix and now names the limit and points to --timeout. None or 0 means no limit.
    • It raises RuntimeError if the websocket drops, or if the kernel reports dead or restarting, instead of waiting forever.
    • The comments in execute_code now describe both timeout behaviors accurately.
  • --timeout help text: "Seconds to wait without output before failing (0 for no limit)".
  • docs/: dated log entries, the options table in 05_run_command.md, and the timeout bullet in 02_execution_and_interactive.md.

Behavior change

  • Code that keeps printing now runs to completion, however long it takes. Its local client stays idle.
  • On 0.8.0 and 0.9.0, code that is silent for longer than --timeout now fails with TimeoutError, as documented, where it used to complete while a CPU core stayed busy. For colab run without --keep, the session is then stopped. Users with long silent phases can raise --timeout, or pass --timeout 0.
  • colab repl passes no timeout, so it now waits until each cell finishes, or until the kernel dies or restarts.

Scope

These paths still go through jupyter-kernel-client and are unchanged:

  • Executions with stdin: colab auth and colab drivemount, each with an explicit timeout.
  • The non-streaming path: colab install, and the /content setup that runs before each command.

colab install passes no timeout, so a long installation can still hit the library's 10-second deadline. Routing it through _execute_streaming would be a natural follow-up.

Tests

  • test_colab_runtime_execute_interactive_with_timeout keeps its assertion: timeout reaches execute_interactive. The call now passes allow_stdin=True, because only that branch still calls execute_interactive.
  • The two tests in test_streaming.py keep their names and assertions. They now use MockKernelClient (tests/mock_kernel.py), which replays scripted kernel messages on a simulated clock. Only time.monotonic, as seen by runtime.py, is simulated.
  • New tests cover:
    • the idle limit in both directions: a 599-second silence completes and a 601-second silence fails, with a limit of 600 s;
    • a 10-minute run that keeps printing and completes;
    • a timeout that counts from the last message, with the --timeout hint in the error;
    • no limit with None or 0;
    • a reply that arrives after the idle status;
    • a reply that never arrives;
    • replies, idle statuses and output from other requests, which are ignored;
    • bounded waits;
    • a connection lost partway through the run;
    • a kernel that dies or restarts;
    • execute called with allow_stdin=False.
  • I checked that the tests catch mistakes in the new code. Removing the idle reset, the kernel-state check, the parent filter, the 0 handling or the connection check each makes a test fail.
  • uv run pytest: 373 passed, 19 skipped. Ruff check, format and import sorting pass on the changed files.

Verification on Colab runtimes

CPU runtimes, with jupyter-kernel-client 0.8.0, 0.9.0 and 1.0.2:

Case --timeout Result
colab run, prints every 5 s for 60 s default (30) Completes on all three versions. The client uses 0.0–0.1% CPU (98–99% before, with 0.9.0)
colab run, silent for 45 s default (30) TimeoutError: Timeout waiting for output: no message from the kernel for 30s. Use --timeout to allow longer silences., exit code 1, session stopped
colab run, silent for 45 s 60 Completes
colab run, silent for 45 s 0 Completes
colab exec on an existing session, prints every 5 s for 60 s default (30) Completes at 0.0% CPU
colab exec, silent for 45 s default (30) TimeoutError, exit code 1
Ctrl-C 40 s into a silent colab run 600 Exits 1.8 s after SIGINT with exit code 130, session stopped

Follow-up

With this change, it is safe to raise the jupyter-kernel-client pin to >=1.0.2. That change also affects tool.uv.sources, so it is not part of this PR.

`--timeout` (default 30 s) is documented as an idle threshold, so that
long-running silent tasks can raise it and code that keeps printing is
never cut off. Runtime.execute_code passed it to jupyter-kernel-client's
execute_interactive, where it is a deadline for the whole run: 0.8.0 and
0.9.0 never enforce it and poll at ~100% CPU once it passes, and 1.0.2
raises TimeoutError even while output flows.

For streaming execution without stdin (run, exec, repl), the runtime now
waits for kernel messages itself, through the websocket client's execute,
get_iopub_msg and get_shell_msg, in blocking slices of up to one second:

- Every message from the execution resets the idle clock; messages from
  other requests are ignored.
- After `timeout` seconds without a message it raises TimeoutError, now
  with a hint to use --timeout. `--timeout 0` (or None) means no limit.
- It raises RuntimeError if the connection drops or the kernel dies or
  restarts, instead of waiting forever.

The --timeout help text and the design docs describe the idle semantics.
Executions with stdin (auth, drivemount) and the non-streaming path
(install, the /content prelude) keep using jupyter-kernel-client.

Fixes googlecolab#183
@hohieuai
hohieuai force-pushed the fix/timeout-idle-limit branch from 7e6cb25 to e717770 Compare October 24, 2026 01:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default --timeout 30 caps the whole run: every longer colab run busy-loops a CPU core today, and fails once jupyter-kernel-client is upgraded

1 participant