Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
.claude/scheduled_tasks.lock
.claude/worktrees/
.superset/
.correctness/

.idea/
/rshell
Expand Down
9 changes: 8 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
.PHONY: build fmt test test_all test_against_bash compliance
.PHONY: build fmt test test_all test_against_bash compliance correctness-scan

TARGET ?= changed
DEPTH ?= quick
CORRECTNESS_SCAN_ARGS ?=

build:
go build -o rshell ./cmd/rshell
Expand All @@ -17,3 +21,6 @@ test_against_bash:

compliance:
RSHELL_COMPLIANCE_TEST=1 go test -v ./tests/ -run TestCompliance -count=1

correctness-scan:
go run ./tools/correctness-scan --target "$(TARGET)" --depth "$(DEPTH)" $(CORRECTNESS_SCAN_ARGS)
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@ The interpreter supports Linux, macOS, and Windows. Some host-inspection builtin

See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, testing, and pull request guidance. Security-sensitive builtin implementation rules live in [docs/RULES.md](docs/RULES.md).

Maintainers can run the AI-assisted, behavior-only correctness scanner locally; see [Correctness scanner](docs/CORRECTNESS_SCANNER.md). It is separate from the repository's security-review tooling and reports one GitHub issue per publishable finding.

## License

[Apache License 2.0](LICENSE)
108 changes: 108 additions & 0 deletions docs/BEHAVIOR_CONTRACTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# rshell Behavior Contracts

These contracts define the user-visible behavior examined by the correctness
scanner. They are compatibility contracts, not security invariants. The scan
must not investigate authorization bypasses, sandbox escapes, secret exposure,
resource-exhaustion attacks, privileged execution, or other threat-model
concerns covered by the repository's security tooling.

## Sources of truth

Apply these sources in order:

1. Explicit accepted design decisions and intentional divergences.
2. `SHELL_FEATURES.md` for the supported rshell surface.
3. User-visible flag, help, and output requirements in `docs/RULES.md`.
4. Bash and GNU behavior for functionality rshell claims to support without a
documented divergence.
5. Builtin help and command descriptions.
6. Existing tests as evidence of intent, but not as final authority.

The current implementation is the subject under examination and is never an
authority merely because the checked-in tests agree with it. When authoritative
sources disagree, report a documentation ambiguity instead of choosing one.

## Contracts

### BEH-001: Supported shell semantics

For supported syntax and ordinary bounded inputs, parsing, expansion,
execution order, and exit-status propagation match Bash unless an intentional
divergence is documented.

### BEH-002: Builtin output

For supported flags and operands, deterministic builtins produce the expected
stdout bytes, including separators, headers, ordering, line endings, and
trailing-newline behavior.

### BEH-003: Errors and status

Ordinary invalid usage produces the documented stderr and exit status. This
contract covers user-visible behavior only; whether blocked functionality can
cross a security boundary is outside the scan.

### BEH-004: Input sources

Standard input, `-`, file operands, multiple files, pipes, and command
substitutions interact as documented and, where applicable, like the reference
command.

### BEH-005: Flag interactions

Supported short, long, combined, repeated, and reordered flags retain their
documented meaning. Attached values, separate values, and `--` termination
follow the documented or reference behavior.

### BEH-006: Shell state

Variables, working directories, `$?`, loops, and subshell isolation have the
documented lifetime and propagation behavior. Inline assignments remain scoped
correctly, and subshell changes do not escape to their parent.

### BEH-007: Formatting and ordering

Deterministic formatting, numeric rendering, sorting, labeling, and record
boundaries match the selected oracle for supported inputs.

### BEH-008: Intentional differences

Documented rshell-specific behavior matches the documentation even when Bash
or a GNU command behaves differently. A reference mismatch is not a defect
when the divergence is explicit and the implementation follows it.

### BEH-009: Surface consistency

Builtin registration, command descriptions, `--help`, `README.md`, and
`SHELL_FEATURES.md` agree about the supported user-visible command and flag
surface.

## Initial scan scope

The first scanner version focuses on deterministic text-processing builtins
and positive shell semantics:

- `cat`, `cut`, `echo`, `grep`, `head`, `printf`, `sed`, `sort`, `tail`,
`test`, `tr`, `uniq`, and `wc`.
- Variables, quoting, expansion, pipes, substitutions, AND/OR lists, loops,
conditionals, and subshells.

Other targets may be requested, but a scan must mark a claim inconclusive when
the target depends on live host state, privileged access, or a security-policy
decision rather than a stable behavioral oracle.

## Excluded security concerns

The correctness scanner must not report findings about:

- `AllowedCommands` or `AllowedPaths` bypasses.
- Remediation or system-service authorization.
- Privileged-helper behavior.
- Secret, environment, process, or telemetry exposure.
- Host-observation trust boundaries.
- Resource-exhaustion attacks or security limits.
- Cross-tenant isolation.
- Side effects occurring before authorization.

Those concerns belong to the security threat pack and its dedicated review
workflows.
111 changes: 111 additions & 0 deletions docs/CORRECTNESS_SCANNER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Correctness scanner

The correctness scanner is a local, AI-assisted review tool for rshell's
user-visible behavior. It uses Codex to inspect one bounded target, generate
temporary tests in a detached worktree, and validate suspected mismatches with
the repository's existing scenario and Bash-comparison infrastructure.

It is intentionally separate from security review. The scanner reads
`docs/BEHAVIOR_CONTRACTS.md` and does not investigate sandbox escapes,
authorization bypasses, secret exposure, or other threat-model concerns.

## Prerequisites

- A clean, committed Git ref to scan. The current checkout may contain
unrelated untracked files; the scanner always examines the committed ref.
- Go and Git.
- Codex CLI authenticated for local use.
- Docker when a Bash comparison is required.
- GitHub CLI authenticated for issue publication.

The scanner uses the Codex CLI's configured default model. Model selection is
not exposed by V0.

## Run locally

Scan changed behavior relative to `origin/main`:

```bash
make correctness-scan TARGET=changed DEPTH=quick
```

Scan one builtin:

```bash
make correctness-scan TARGET=cat DEPTH=standard
```

Issue publication is enabled by default. For a calibration or offline run:

```bash
make correctness-scan TARGET=cat CORRECTNESS_SCAN_ARGS=--no-publish
```

Local runs print each scanner phase. When the investigation begins, the
scanner prints an exact `tail -f` command for following the full Codex log in
another terminal without flooding the command's normal output. Use `--quiet`
to suppress the phase messages while still writing the log files:

```bash
make correctness-scan TARGET=cat CORRECTNESS_SCAN_ARGS="--no-publish --quiet"
```

The underlying command also accepts `--ref`, `--base-ref`, `--repo`, and
`--output-dir`:

```bash
go run ./tools/correctness-scan \
--target shell:variables \
--depth quick \
--ref HEAD \
--no-publish
```

## Artifacts

Each run writes to the gitignored `.correctness/runs/<run-id>/` directory:

- `prompt.md` — the exact investigator prompt.
- `schema.json` — the structured output contract passed to Codex.
- `codex.stdout.log` and `codex.stderr.log` — complete Codex process output.
- `findings.json` — the structured final response.
- `report.md` — the human-readable scan report.
- `issues/*.md` — one publication preview per publishable finding, including
when `--no-publish` is used.
- `issue-links.md` — issues created, updated, or reopened by the run.

The temporary Git worktree is removed after the scan unless
`--keep-worktree` is supplied for debugging.

## Publishing rules

The scanner publishes one issue per finding for these classifications:

- Confirmed correctness defect.
- Actionable documentation ambiguity.
- Actionable missing coverage.

Inconclusive observations remain in the local report. A stable fingerprint in
each issue body prevents duplicate issues. A repeat finding comments on the
existing issue; if the issue was closed, the scanner reopens it first.

The planned V1 automation will invoke this same command and publication path
on an Actions runner. Only scheduling and credential sourcing move into the
workflow, so local and automated scans use identical finding classifications,
validation, fingerprints, and issue bodies.

Use `--no-publish` whenever a run should not mutate GitHub. Publishing uses the
developer's existing `gh` authentication and never passes GitHub credentials
to Codex or generated test programs.

## Isolation and evidence

Codex receives workspace-write access only to the detached temporary worktree.
It may add temporary scenario or Go tests, but it must not modify production
code, commit, push, or create issues. Issue publication happens afterward in
the parent scanner process.

A confirmed defect must include an exact reproducer, expected and actual
stdout/stderr/status, at least two successful reproductions, source references,
and implementation references. Claims without that evidence are rejected or
reported as inconclusive.
23 changes: 23 additions & 0 deletions interp/expansion_limits_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"bytes"
"context"
"errors"
"fmt"
"strings"
"sync/atomic"
"testing"
Expand Down Expand Up @@ -94,6 +95,28 @@ func TestExpandedArgumentCountLimitAllowsBoundary(t *testing.T) {
}
}

func TestNonWhitespaceIFSFieldCountLimit(t *testing.T) {
// Empty fields consume argument slots even though they consume no bytes.
for _, count := range []int{MaxExpandedArgumentsPerCommand, MaxExpandedArgumentsPerCommand + 1} {
t.Run(fmt.Sprint(count), func(t *testing.T) {
stdout, stderr, err := runExpansionScript(t, "IFS=:\ntrue $X\necho done",
Env("X="+strings.Repeat(":", count)),
AllowedCommands([]string{"rshell:true", "rshell:echo"}),
)
if count == MaxExpandedArgumentsPerCommand {
if err != nil || stderr != "" || stdout != "done\n" {
t.Fatalf("boundary: stdout=%q stderr=%q err=%v", stdout, stderr, err)
}
return
}
requireExitStatus(t, err, 1)
if stdout != "" || !strings.Contains(stderr, "expansion exceeds maximum field count") {
t.Fatalf("overflow: stdout=%q stderr=%q", stdout, stderr)
}
})
}
}

func TestExpandedArgumentByteLimit(t *testing.T) {
value := strings.Repeat("x", 1<<20)
script := "true " + strings.TrimSpace(strings.Repeat("$X ", 11))
Expand Down
Loading