[matt-dz]

What does this PR do?

• Preserve empty arguments from leading and consecutive non-whitespace IFS separators, including mixed whitespace and adjacent expansions. A trailing separator terminates the last field without adding an extra argument.
• Split these expansions locally while retaining upstream brace expansion and globbing, evaluating substitutions once in source order, and enforcing the existing argument and byte limits.
• Replace the scenario that expected the upstream defect with Bash-compatible regression coverage, including quoting, substitutions, globbing, and the empty-field argument limit.

Motivation

With IFS=:; x=':a::b:', unquoted $x dropped two empty arguments and shifted the remaining argument list. This reproduces correctness scan 0c03c974-e3c3-4f39-aca6-1a72999952ef against the current code and conflicts with the documented Bash-compatible behavior. The pinned upstream splitter coalesces all IFS separators; there is no newer released dependency version to adopt.

Testing

• The corrected regression scenario failed before the implementation change and passes afterward. All field-splitting scenarios and the argument-limit boundary test pass using Go 1.26.3.
• Native Bash matches all 46 eligible field-splitting scenarios byte-for-byte and 1,008 additional separator/adjacent-expansion/quoting combinations. The new scenario expectations were also checked directly against Bash.
• The interpreter suite and all shell scenarios pass with the race detector. The broader go test -race ./interp ./tests -count=1 -timeout 300s run has 10 unrelated ping scenario failures because this sandbox resolves no-such-host-xyzzy.invalid to a responding address; a representative failure also reproduces with the original binary.
• The required Docker Bash comparison suite was invoked but skipped after pulling debian:bookworm-slim returned HTTP 502.

Checklist

• Tests added/updated
• Documentation updated (if applicable)

Correctness scan: 0c03c974-e3c3-4f39-aca6-1a72999952ef

Finding fingerprint: eaa13a504500472ba49fe0b883d15a16c9dba438a0f7c2c69bceb2582c7a156a

---

PR by Bits - View session in Datadog

Comment @datadog-staging to request changes

[datadog-staging]

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @datadog-staging to request changes

[julesmcrt]

Reviewed f8bbe654.

The field-splitting fix is correct and I'd like to see it land — but this PR still carries the correctness-scanner scaffolding commits (b0610e8, d61beb8, cc90105, 514d1cb), which duplicate the sibling scanner PRs and include the removal of the Codex --sandbox workspace-write pin next to --approve-for-me in tools/correctness-scan/main.go (invokeCodex). That's the same P1 the #629 review already found: an auto-approving, unsandboxed Codex child process. The scanner is already covered by that review; please rebase these commits out of this PR (or restore the pin there first) — with just the IFS fix remaining, this is an approve from me.

On the fix itself, the verification summary:

• Current main (post-#695) still drops empty fields: across a 67-case harness, 28 cases diverge between this branch and main — 26 are split-semantics cases where main coalesces non-whitespace IFS delimiters (a::b → 2 fields instead of a,``,b; `:a::b:` → 2 instead of 4; `::::` → 0 instead of 4; leading/mixed/multibyte/subshell/pipe variants likewise), one pair is the field-count cap, one is a sandbox-error exit propagation. On every split-semantics case the branch matches a bash 5.3 oracle exactly, including the subtle ones: a lone `:` is one field, `::` is two, `a:` is `[a]` with no trailing empty, `a::` is `[a]` plus an empty, and `:a::b:` is four fields.
• Merging onto current main is clean, and the interactions are good: #695's field-source estimator and the yield-time field cap still bound the new path — empty fields consume cap slots, so 16,385 delimiter-generated fields now fail with "expansion exceeds maximum field count" where main silently proceeds (the new TestNonWhitespaceIFSFieldCountLimit pins this; it passes on the merge, as do the new/updated scenario yamls). Redirect targets, which main routes through the same collector, also get more bash-correct: IFS=:; x=':a::'; cat < $x is "ambiguous redirect" (3 fields, like bash) instead of opening a. Tilde rejection still holds through the new path (IFS=: + echo {,}~root is still refused), and a tilde produced by an expansion result stays unexpanded, as in bash.
• Security-wise the change is in the safe direction: an empty leading field now fails closed as a command name (127, "command not allowed") where main silently returns 0; empty argument fields can't alter exact-name command authorization; byte/field/brace/glob-panic bounds all still apply; I found no new panic or accounting surface.

Two minor, non-blocking items: (1) interp/field_splitting.go:117-118 — when IFS contains a literal U+FFFD, an invalid-UTF-8 byte in an unquoted expansion result matches utf8.RuneError from range and is treated as a delimiter (IFS=$(printf '\xef\xbf\xbd'); x=$(printf 'q\xffr'); for w in $x … splits where bash would not). Obscure, no safety impact, but worth a DecodeRuneInString-size check or a note. (2) The read builtin still mishandles non-whitespace IFS independently of this change (IFS=:; printf 'a::b\n' | read A B C yields three empty variables where bash gives a,``'',b); it's identical on main, so not a regression — but since the PR fixes the same class of bug for word expansion, a follow-up issue would be good.

One process note: consecutive_nonwhitespace_ifs.yaml drops its skip_assert_against_bash and now asserts bash's output — the expectations match bash 5.3 locally. The Docker-based scenario comparison couldn't be run in this environment; the scenario expectations were cross-checked against a local bash 5.3 oracle for every case above.