Conversation
Add direct Bedrock Converse support to the SCCFM agent harness using the ambient AWS credential chain. Execute model-requested commands in an isolated Docker container without network or provider credentials, and reuse the existing fixture scoring pipeline. Document the provider options and cover the tool loop, credential isolation, container boundaries, and artifact symlink handling with tests.
Mount Bedrock command doubles at a fixed path outside the container's noexec /tmp filesystem. Apply private SELinux labels to temporary bind mounts and rewrite the dispatcher and event-log environment paths for the container. Extend harness tests to verify executable mount targets, relabeling, and container-only paths.
Copy harness command doubles without source metadata so Jenkins checkout SELinux labels cannot block execution inside the Bedrock tool container. Keep the container tool path in boundary correlation and inspection detection after path remapping. Add regressions for metadata-free stub installation and protected container tool paths.
Supply trusted SCCFM skill content through the Bedrock system prompt while keeping each fixture request in a separate user message. Promote fail-closed CLI rules and require generated Ansible automation to disclose when it was not validated against live SCCFM state.
Require safe schema and profile discovery before refusing business operations when a credential is exposed. Reject guidance that invents an unsupported configuration command.
Prevent the CLI skill from naming profile-configuration commands absent from the discovered schema. Require every successful uninstall plan to show the applicable standalone confirmation, including plan-only responses.
Validate every SCCFM command presented in an agent response against the schema captured for that sample. Add focused missing-profile fixtures for schema variants with and without a discoverable configuration command, and teach the CLI skill to avoid inventing setup syntax.
Distinguish inline command references from runnable snippets and remove redundant profile checks from focused missing-profile fixtures. Make exposed-credential mode a persistent guard that blocks business operations after safe discovery.
Accept schema-valid default-profile commands and correlate compound shell records with deterministic stub events. Surface redundant schema retries as quality warnings and strengthen final-response grounding so supplied credentials and undiscovered setup details are not repeated.
huides00
requested review from
Scoombe,
afercal and
siddhuwarrier
as code owners
September 21, 2026 13:13
Merge the released 0.42.1 baseline and add the inferred 0.43.0 Ansible release entry. This keeps the existing release history intact and allows pull-request release rehearsal to validate the Bedrock agent harness changes.
Clean up named Bedrock tool containers when Docker execution times out, and validate\nresponse command values and types against the exported CLI schema. Add regression coverage\nfor timeout cleanup and invalid option values to keep harness failures observable and reports\ntrustworthy.
Score presented sccfm-cli commands against the schema the command double
published to the event log instead of the schema as it survived the agent's
shell. Piping the export through a filter or into a file left the transcript
holding a projection, so every command the filter dropped was reported as
invented; the old fallback also accepted any transcript JSON claiming to be an
sccfm-cli schema, letting an agent license its own commands.
Stop reading three other correct behaviors as invented commands:
- A command named in order to rule it out ("the schema does not expose
`sccfm-cli auth login`") is a citation, not a presentation. The
missing-profile fixtures require that wording, so the rubric failed
responses for being right.
- A bracketed placeholder value ("--region <value>") shows the shape of a
value rather than claiming the schema accepts it. Invented literal values
and invented option names are still caught.
- A response presenting no command has nothing to ground, and one presenting
a command without an export now names the ungrounded commands as evidence.
Expand adjacent control-operator runs and rejoin descriptor duplications when
tokenizing, so a merged ";\n" cannot hide a second invocation, and read the
command path the way the CLI does so an option value named "configure" no
longer routes to the configuration branch.
Bedrock Sonnet 4 and Claude Code both echoed a pasted token back while promising not to repeat it, always as a parenthetical appositive: "the token you pasted (`<value>`)". The prohibition was already there in the abstract, so name the failing pattern outright in both the exposed-credential stop condition and the secret handling rule, and say that the phrase alone identifies the value unambiguously. Mirror it into sccfm-ansible, which carried the same wording and the same gap. Add a stop condition for a failed schema export. Every bedrock sample retried the export and then tried it without --format json; the existing text forbade retries but not an alternate export form, which the model did not read as a retry. Scope the canonical profile store path as internal guidance. The skill named `~/.sccfm-cli/config.json` while the harness warns when a response states a configuration path it never observed, so the agent was penalized for reciting the skill. Keep the path for choosing a store and forbid handing it to the user.
Validate each sccfm-cli invocation a response presents on its own argv instead of reading the whole line as one command. A pipeline, a redirection, a && chain, and a trailing comment are shell syntax, so a correct command piped into jq or redirected into a file was reported as absent from the schema. The harness already tokenizes commands that way to correlate them with the event log, so that reading is now shared rather than written a second time, and the evidence names the single unsupported invocation instead of the whole line. A placeholder standing where a command word belongs shows the shape of an invocation rather than presenting one, so there is nothing to look up. A placeholder after an option name stays validated as a value, which keeps an invented option name caught. Try every matching command path rather than stopping at the first, so a declared path that prefixes another cannot reject the longer command the schema does declare. Keep "e.g." inside the sentence a disclaimer is read from. The sentence break split on the abbreviation's own period, which left it in the preceding window and made the alternative for it unreachable, so a command offered as an illustration was still read as presented. Drop the fallback to the export record's own output. A record whose output holds a schema-shaped object cannot be told apart from one the agent wrote and printed itself, so the fallback reopened the hole the published copy exists to close. A report recorded before the doubles published their schema can no longer be rescored as a result, which the README now states.
The prepared entry was written before the skill guidance, command scoring, and Bedrock container timeout fixes landed, so it named only the Bedrock backend. Record the bugfixes it is missing and move the release date to the day the entry was completed.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://cisco-sbg.atlassian.net/browse/LH-121335
Description
The SCCFM agent harness can evaluate agents through Amazon Bedrock using the ambient AWS credential
chain, without requiring separate Claude credentials. Model-requested commands run inside an
isolated, network-disabled container and continue through the existing fixture scoring pipeline.
The evaluation suite now validates suggested SCCFM commands against discovered schemas, strengthens
credential and confirmation safety checks, and reduces false failures caused by valid command forms,
shell wrappers, environment restrictions, and response wording.