test(fixtures): add self-contained openai-default test system YAML by Karn2898 · Pull Request #36 · extra-org/extra

Karn2898 · 2026-07-20T15:38:09Z

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example. - test_system.yaml defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:. A comment makes explicit that the key is supplied via the environment, never in YAML. - Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline (no LLM key required).

AmitAvital1 · 2026-07-20T19:13:33Z

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

Karn2898 · 2026-07-21T19:09:08Z

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

AmitAvital1 · 2026-07-21T19:14:31Z

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

Karn2898 · 2026-07-24T06:45:20Z

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

AmitAvital1 · 2026-07-24T08:28:57Z

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

Karn2898 · 2026-07-26T22:15:51Z

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

AmitAvital1 · 2026-07-27T18:41:56Z

@@ -0,0 +1,111 @@
+from __future__ import annotations


maybe lets call this file utils?

AmitAvital1 · 2026-07-27T18:43:18Z

+
+
+FIXTURE_DIR = Path(__file__).resolve().parent
+FIXTURE_SPEC = FIXTURE_DIR / "test_system.yaml"


maybe lets do it injectable? think about we will added more yamls for tests in your infra

That's a good point . I have made changes ,just need to push

AmitAvital1 · 2026-07-27T18:51:35Z

+    temperature: 0.0
+
+execution:
+  max_iterations: 10


what about test for all of this configurations?

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

AmitAvital1

Nice work. can we also test generate?

AmitAvital1 · 2026-07-27T18:54:30Z

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

- test_system.yaml: small openai-default fixture with auto:true on tool-using agent - prompts/ + plugins/: implemented stubs so agentctl validate passes offline - utils.py: shared FakeChatModel, fake_model_factory, load_test_system, FakeEngine - test_infra.py: 5 core tests covering validation, build, routing, tools, resolvers - test_engine_flow.py: dedup FakeChatModel into tests.fixtures.utils Closes the shared-module import collision with an autouse cleanup fixture.

Karn2898 · 2026-07-27T20:11:52Z

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

AmitAvital1 · 2026-07-27T20:32:41Z

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

Asaf-prog assigned Asaf-prog and Karn2898 and unassigned Asaf-prog Jul 21, 2026

Asaf-prog requested review from AmitAvital1 and Asaf-prog July 21, 2026 16:39

test(fixtures): avoid sys.modules conflict by removing shared.py

1931c1a

AmitAvital1 reviewed Jul 27, 2026

View reviewed changes



		FIXTURE_DIR = Path(__file__).resolve().parent
		FIXTURE_SPEC = FIXTURE_DIR / "test_system.yaml"

Uh oh!

Conversation

Karn2898 commented Jul 20, 2026

Summary

Verification

Uh oh!

AmitAvital1 commented Jul 20, 2026

Uh oh!

Karn2898 commented Jul 21, 2026

Uh oh!

AmitAvital1 commented Jul 21, 2026

Uh oh!

Karn2898 commented Jul 24, 2026

Uh oh!

AmitAvital1 commented Jul 24, 2026

Uh oh!

Karn2898 commented Jul 26, 2026 via email

Uh oh!

AmitAvital1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Karn2898 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

AmitAvital1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Karn2898 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

AmitAvital1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Karn2898 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

AmitAvital1 left a comment

Choose a reason for hiding this comment

Uh oh!

AmitAvital1 commented Jul 27, 2026

Uh oh!

Karn2898 commented Jul 27, 2026

Uh oh!

AmitAvital1 commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants