Repository navigation
Harness agentic testing - #534
Draft
chrishagglund-ship-it wants to merge 8 commits into
Draft
chrishagglund-ship-it wants to merge 8 commits into
chrishagglund-ship-it wants to merge 8 commits into
Conversation
Theorized proof of concept for exercising the agent runtime from the harness. Not yet run against a real server; committed so we can test how this behaves in reality and iterate. Agentic testing is off by default (HARNESS_AGENTIC_TESTING_ENABLED), so the existing deployment behaves as before. - main.py: split into modules with prepare/start/stop/wait so every module forks its worker processes before any governor thread starts - sdk_testing.py: the existing simulated-task testing, moved unchanged - harness_env.py: env helpers shared by both modules - agentic/: a storyteller agent writes one sentence per turn and calls a critic sub-agent; its stop_when worker ends the loop when the critic's JSON verdict says the story meets a randomly invented concept, or max_turns ends it. A governor starts runs on an interval and swaps in a new concept every few runs by redeploying under the same name. - try_agentic.py: runs one plan or one story locally without the governors, cancelling the run on Ctrl-C or timeout Ideas to test: - whether the critic's verdict reaches stop_when through the LLM's message history, and what shape it arrives in - whether redeploying under the same name replaces version 1 in place - behaviour on Orkes 5.6.0-rc4 (OSS 3.32.1) vs OSS 3.33.0 - record/replay through mock/mockLLM with a fixed seed (OSS 3.33.0+) - real token cost per story and how many turns concepts take Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Print a link to the execution in the UI when a run starts and in the
final report. The UI address is the server URL without /api, or
--ui-url / HARNESS_UI_URL when they differ.
- --open opens that link in the browser.
- --verbose shows the SDK's own INFO logs (not the HTTP library's, which
can print request headers carrying the auth token). Added to see which
workers the server asks the SDK to run: on Orkes 5.6.0-rc4 a run started
from the agent object never got a stop_when worker and waited forever.
- story_prompt() no longer doubles the period after settings that end in
one ("at 3 a.m..").
Still untested against a real server end to end.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stop_when decides whether the storyteller's loop ends, and that decision
depends on the result of a tool call: the critic's verdict. As compiled,
stop_when reads the LLM call's input messages, a snapshot taken before
that turn's critic call, so it sees each verdict one turn late and the
loop runs one extra, most expensive storyteller call after approval.
- agentic/stop_wiring.py: after deploy, re-point stop_when's messages
input at ${workflow.variables._last_tool_results}, which already holds
every tool result so far and is written after the critic and before
stop_when. Edits the raw definition JSON so no field is lost, refuses
to write a definition without its agent markers in metadata, and
re-reads it to confirm. Must be re-applied after every deploy; the
proper fix is in the server's compiler.
- try_agentic.py run --stop-same-turn applies it before starting.
- agentic/run_report.py: build the report from the execution record, not
streamed events (Orkes 5.6.0-rc4 streams no tool-call events): each
turn's sentence and verdict, tokens for storyteller and critic, which
stop_when wiring the run used, and the story the critic approved
rather than the storyteller's final reply, which can add commentary.
Reads go through call_api(response_type="object") because the SDK's raw
mode (_preload_content=False) crashes on its httpx transport.
Checked against a fake server through the real SDK and against saved
executions from real runs; not yet run on a cluster.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The harness governor now applies the same stop_when edit as try_agentic.py run --stop-same-turn, so stop_when sees the critic's verdict on the approving turn and each run saves one storyteller call. HARNESS_AGENTIC_STOP_SAME_TURN=false turns it off. - agentic/agentic_governor.py: every deploy undoes the edit, so it is due after the initial serve() and after each concept rotation, and is applied before the next run starts. If it fails (refused, HTTP or connection error), the governor logs a warning and does not start that run; it tries again once at the next interval, without retrying in a loop. - agentic/agentic_testing.py: reads the env var (default true) and hands the governor a MetadataResourceApi when it is on. - agentic/stop_wiring.py, README: why the edit reuses the messages input (the Python SDK's stop_when worker drops any other input), and the proper fix: single-agent stop_when can't see this turn's tool results; add a toolResults input in the server's compiler and expose it in each SDK's stop_when context. - README: describe running against Orkes only, where the provider key is stored in an AI integration and the model is <integration name>/<model>. Tested so far only with try_agentic.py against Orkes 5.6.0-rc4; drop the untested Conductor OSS instructions and the mock/mockLLM claim. Governor behavior checked with a fake runtime and a fake rewire; the harness mode has not been run on a cluster yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The governor started runs and never looked at them again, so a run that
failed (for example, a wrong integration name) was silent in the log.
- agentic/agentic_governor.py: between starts, check the last run every
15 seconds and report it as soon as it has finished: status, turns,
the turn the critic approved and whether the run stopped on it, which
stop_when wiring it used, critic calls, storyteller tokens, a link,
and the story the critic approved. Checks that find it still running
are silent. If it hasn't finished by the next start, that start says
so once and stops tracking it. On stop, name a run not yet reported.
- The story is read from the execution, not the workflow output. A run
that stops on the approving turn ends on a critic call, and the
server's synth_output fallback then drops the tool-call inputs: its
JSON.stringify serializes Java maps as {}. Reproduced with the
server's GraalJSEvaluator.
- agentic/run_report.py: add the failure reason; execution_url() moves
here from try_agentic.py.
- Remove --ui-url and HARNESS_UI_URL: links are always the server URL
without /api, which is where Orkes serves the UI.
- agentic/stop_wiring.py, README: name the proposed stop_when input
tool_results, as the SWARM handoff checks already do.
- README: explain why the concept and the setting are separate; add a
Known Issues section for the server, SDK and CLI problems found so
far; note testing on Orkes 5.6.0-rc4 and rc6 (both OSS 3.32.1).
Checked against a run on Orkes 5.6.0-rc6 and with fakes. SDK testing
code is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
python-agent-harness-worker runs only the agentic module, next to the SDK harness deployment. One replica with the Recreate strategy: each pod runs its own agentic governor, and two would overwrite each other's python_storyteller and python_story_critic definitions mid-run. The model names the chatbots AI integration on certification-aws. PYTHONUNBUFFERED=1 so the stop_when worker's progress lines, printed from a spawned child process, reach the pod log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A run in progress when the agentic harness stops can't finish correctly: its stop_when worker is gone, and the next harness deploys a new concept, so the run calls a critic with different criteria from its next turn on (the critic is looked up by name on every call). - agentic/agentic_governor.py: on stop, report the last run if it has just finished, otherwise terminate it. - agentic/agentic_testing.py: on startup, before serve() deploys, terminate any python_storyteller run still RUNNING, e.g. one left by a crash. - agentic/story_agents.py: story_is_complete's docstring now describes both stop_when wirings. - README: describe both, and run only one agentic harness per cluster. Checked with fakes; not yet run on a cluster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.