parallel-checkout: fix stack buffer overflow in Windows poll() with many workers (#6395)

### Symptom

On Windows, `git checkout` and `git reset --hard` can abort with

```
*** stack smashing detected ***: terminated
```

and exit code `0xC0000409` (`STATUS_STACK_BUFFER_OVERRUN`). This is
memory
corruption, not a normal error. The process dies before Trace2 writes
its log,
so nothing shows up in a trace. A `.git/index.lock` is left behind.

It happens when `checkout.workers` is large, or when it is `0` (meaning
"use
`online_cpus()`") on a machine with many logical processors.

### Mechanism

`gather_results_from_workers()` in `parallel-checkout.c` polls one pipe
per
checkout worker:

```c
CALLOC_ARRAY(pfds, num_workers);
...
poll(pfds, num_workers, -1);
```

Windows has no native `poll()`, so `compat/poll/poll.c` emulates it with
`MsgWaitForMultipleObjects()`. It collects one handle per polled
descriptor in a
fixed stack array:

```c
HANDLE h, handle_array[FD_SETSIZE + 2];   /* 64 + 2 = 66 entries */
...
handle_array[nhandles++] = h;             /* no bounds check */
...
handle_array[nhandles] = NULL;            /* sentinel, no bounds check */
```

`FD_SETSIZE` is the Winsock default 64, and nothing in the build
overrides it.
`run_parallel_checkout()` clamps `num_workers` only against the number
of files,
never against the array size or the Windows wait limit. A high worker
count
therefore writes past the end of the array and smashes the stack.

Sockets are not involved: they are multiplexed onto a single event
through
`WSAEventSelect`, so only non-socket descriptors consume a slot.

### Why 62, and where the limit lives

Two of the wait slots are never available for descriptors:

* `compat/poll` uses index 0 for its own event object.
* `QS_ALLINPUT` adds the thread message queue as an implicit wait
object. The
  code confirms this, because it reports the message queue as
  `WAIT_OBJECT_0 + nhandles`.

So `nhandles + 1 <= MAXIMUM_WAIT_OBJECTS`, which gives at most
`MAXIMUM_WAIT_OBJECTS - 2` = 62 descriptors.

That value is defined once, as `POLL_MAX_DESCRIPTORS` in
`compat/poll/poll.h`
alongside the `poll()` declaration it constrains, and both callers clamp
against
it. `compat/posix.h` defines it to `INT_MAX` where a native `poll()` is
used, so
the callers need no `#ifdef`.

### Why the array cannot simply be enlarged

`MAXIMUM_WAIT_OBJECTS` is a kernel limit, not a header convenience.
Passing more
handles fails with `ERROR_INVALID_PARAMETER`. Growing the array would
only turn
memory corruption into a functional failure. Support for more
descriptors needs
a wait tree (helper threads each waiting on at most 62 handles) or
completion
ports, which is out of scope here.

### Why it surfaced in 2.54

`parallel-checkout.c` and `compat/poll/poll.c` are unchanged between
2.53 and
2.54. Only `online_cpus()` changed:

| Version | API | Result |
| --- | --- | --- |
| 2.53 | `GetSystemInfo()` | processors in the current processor group
only; a group holds at most 64 |
| 2.54+ | `GetLogicalProcessorInformationEx()` | true system-wide
logical processor count |

The old API could never report more than 64, so the array always fit.
That
ceiling was accidental, not deliberate. The `online_cpus()` change is
correct and
must stay; it only exposed a latent bug.

### The changes

1. **`compat/poll: do not collect more handles than the wait supports`**
— defines
`POLL_MAX_DESCRIPTORS` (62) next to the `poll()` declaration and refuses
to
collect beyond it, returning `EINVAL` instead of appending past the end
of the
array. Two preprocessor assertions tie the constant to
`MAXIMUM_WAIT_OBJECTS`
and to the size of `handle_array`, so they cannot drift apart. `poll()`
is now
   memory-safe for every input.

The error path also undoes the `WSAEventSelect()` registrations made
earlier in
the same call. The loop that normally does that runs after the wait, so
returning early would otherwise leave those sockets bound to `poll()`'s
static
event object and let later socket activity disturb an unrelated
`poll()`.

The limit is on the handles actually collected, **not** on `nfd`. Those
are
different: a descriptor only takes a handle when it is non-negative, is
not a
socket, and has no events pending yet. Callers routinely pass sparse
arrays —
`run_processes_parallel()` sizes its `pollfd` array to the configured
job count
and leaves the unused slots at `fd = -1`. An earlier revision of this PR
rejected a large `nfd` instead, which broke `t7406`
(`submodule.fetchJobs 67`,
with only a handful of live pipes) with `fatal: poll: Invalid argument`.

   On platforms with a native `poll()` there is no such limit, so
   `POLL_MAX_DESCRIPTORS` is `INT_MAX` and callers can clamp against it
   unconditionally.

2. **`parallel-checkout: limit worker count to what poll() can wait
on`** — clamps
`num_workers` to `POLL_MAX_DESCRIPTORS` in `run_parallel_checkout()`,
the single
choke point before the workers start. The clamp is silent: fewer workers
is
correct, and a warning would fire on every checkout on a large machine.
Also
documents the cap, since `checkout.workers` was described as using one
worker
   per logical core with no upper bound.

3. **`run-command: limit concurrent children to what poll() can wait
on`** — the
same limit applied to the other `poll()` fan-out. `pp_buffer_io()` polls
one
pipe per child sending output and a second per child being fed on stdin,
and
`fetch.parallel` / `submodule.fetchJobs` / `hook.jobs` all accept a high
value
(or `0`, meaning `online_cpus()`). Without this, change 1 would convert
the old
stack smash on that path into a hard `die_errno("poll")`. Only
*concurrency* is
limited; the configured maximum still sizes the arrays and is still
reported by
   the trace, so the total number of tasks run is unchanged.

### Reproduction

No clone, no special hardware, about 10 seconds. A many-core machine is
not
required: a positive `checkout.workers` is used verbatim, and
`online_cpus()` is
consulted only when the value is `0` or less.

```powershell
# Use a NEW directory every attempt (see the note on timing below).
$repo = "C:\tmp\poll-repro-$(Get-Random)"
New-Item -ItemType Directory -Force -Path $repo | Out-Null
Set-Location $repo

git init -q -b main .
git config user.email repro@example.com
git config user.name  repro
git config checkout.workers 200
git config checkout.thresholdForParallelism 1

New-Item -ItemType Directory -Force -Path dir | Out-Null
1..400 | ForEach-Object { Set-Content -Path "dir\f$_.txt" -Value "base $_" -NoNewline }
git add -A; git commit -qm base

git checkout -qb other
1..400 | ForEach-Object { Set-Content -Path "dir\f$_.txt" -Value "changed $_ padding padding padding" -NoNewline }
git commit -qam changed

git checkout -q main
Write-Host "exit=$LASTEXITCODE"
```

Before the fix, on a 12-core machine, this crashed 3 out of 3 runs with
`exit=-1073740791` (`0xC0000409`) and left `.git/index.lock` behind.
After the
fix it exits 0 on 3 out of 3 runs, with the files correctly updated. A
`checkout.workers 16` checkout still works, as before.

### The crash is not deterministic

`poll()` only appends a descriptor when the worker's pipe has no data
ready yet,
so `nhandles` reflects the workers pending at that instant, not the
workers
spawned. On warm cache, pipes answer immediately and few workers stay
pending.
Measured before the fix:

| workers | result |
| --- | --- |
| 16, 64, 65, 70, 72, 74, 76, 78 | pass |
| 80 | crashed once, then passed 3 times |
| 200, repeated checkouts in the same repo | passed 4 times |
| 200, fresh repository each run | crashed 3 of 3 |

The first out-of-bounds write happens at 65 descriptors by arithmetic,
but the
corruption does not reliably reach the stack cookie until well past
that. The
corruption is real from 65 onward whether or not it crashes. That is why
the fix
targets the contract (62), not the observed crash point.

For the same reason, the added test asserts the **effective worker
count** rather
than a crash. `test_checkout_workers` counts the workers actually
spawned from a
TRACE2 log, so the test verifies the clamp took effect (62) instead of
merely
checking that the checkout did not crash. It is `MINGW`-gated, because
that is the
only platform where the cap applies.

### Testing

* New test in `t/t2080-parallel-checkout-basics.sh`. `t2080`, `t2081`,
`t2082`,
  `t0061`, `t5526` and `t7406` all pass on Windows.
* Manual verification with the reproduction above, plus a
low-worker-count
  regression check.

### Workaround for affected users

```
git config checkout.workers 16
```

Any value at or below 62 avoids the overflow. No downgrade is needed.

Author: dscho

Documentation/config/checkout.adoc

@@ -30,6 +30,11 @@ commands or functionality in the future.
 	all commands that perform checkout. E.g. checkout, clone, reset,
 	sparse-checkout, etc.
 +
+On Windows the number of workers is capped at 62, because the `poll()`
+emulation cannot wait on more worker pipes than that. A higher configured
+value, including the logical core count on a machine with many cores, is
+silently reduced to the cap.
++
 NOTE: Parallel checkout usually delivers better performance for repositories
 located on SSDs or over NFS. For repositories on spinning disks and/or machines
 with a small number of cores, the default sequential checkout often performs

Makefile

@@ -1553,6 +1553,7 @@ CLAR_TEST_SUITES += u-odb-inmemory
 CLAR_TEST_SUITES += u-oid-array
 CLAR_TEST_SUITES += u-oidmap
 CLAR_TEST_SUITES += u-oidtree
+CLAR_TEST_SUITES += u-poll
 CLAR_TEST_SUITES += u-prio-queue
 CLAR_TEST_SUITES += u-reftable-basics
 CLAR_TEST_SUITES += u-reftable-block

builtin/fetch.c

@@ -2351,6 +2351,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
 			.tr2_label = "parallel/fetch",
 
 			.processes = max_children,
+			.no_stdin_pipe = 1,
 
 			.get_next_task = &fetch_next_remote,
 			.start_failure = &fetch_failed_to_start,

builtin/submodule--helper.c

@@ -2919,6 +2919,7 @@ static int update_submodules(struct update_data *update_data)
 		.tr2_label = "parallel/update",
 
 		.processes = update_data->max_jobs,
+		.no_stdin_pipe = 1,
 
 		.get_next_task = update_clone_get_next_task,
 		.start_failure = update_clone_start_failure,

compat/poll/poll.c

@@ -303,6 +303,43 @@ compute_revents (int fd, int sought, fd_set *rfds, fd_set *wfds, fd_set *efds)
 }
 #endif /* !MinGW */
 
+#ifdef WIN32_NATIVE
+/* POLL_MAX_DESCRIPTORS descriptors, plus hEvent and the QS_ALLINPUT message
+   queue, must fit in one MsgWaitForMultipleObjects call, and the collected
+   handles plus the NULL sentinel must fit in handle_array.  */
+#if POLL_MAX_DESCRIPTORS + 2 > MAXIMUM_WAIT_OBJECTS
+#error POLL_MAX_DESCRIPTORS exceeds MAXIMUM_WAIT_OBJECTS
+#endif
+#if POLL_MAX_DESCRIPTORS + 2 > FD_SETSIZE + 2
+#error POLL_MAX_DESCRIPTORS does not fit in handle_array
+#endif
+
+/* Undo the WSAEventSelect() calls made for the first NFD descriptors.  */
+static void
+reset_socket_events (struct pollfd *pfd, nfds_t nfd)
+{
+  nfds_t i;
+
+  for (i = 0; i < nfd; i++)
+    {
+      HANDLE h;
+
+      if (pfd[i].fd < 0)
+	continue;
+      if (!(pfd[i].events & (POLLIN | POLLRDNORM | POLLOUT | POLLWRNORM |
+			     POLLWRBAND | POLLPRI | POLLRDBAND)))
+	continue;
+
+      h = (HANDLE) _get_osfhandle (pfd[i].fd);
+      if (h == NULL || h == INVALID_HANDLE_VALUE)
+	continue;
+
+      if (IsSocketHandle (h))
+	WSAEventSelect ((SOCKET) h, NULL, 0);
+    }
+}
+#endif
+
 int
 poll (struct pollfd *pfd, nfds_t nfd, int timeout)
 {
@@ -504,7 +541,16 @@ poll (struct pollfd *pfd, nfds_t nfd, int timeout)
 	     bits for the "wrong" direction. */
 	  pfd[i].revents = win32_compute_revents (h, &sought);
 	  if (sought)
-	    handle_array[nhandles++] = h;
+	    {
+	      /* hEvent occupies handle_array[0].  See POLL_MAX_DESCRIPTORS.  */
+	      if (nhandles > POLL_MAX_DESCRIPTORS)
+		{
+		  reset_socket_events (pfd, i);
+		  errno = EINVAL;
+		  return -1;
+		}
+	      handle_array[nhandles++] = h;
+	    }
 	  if (pfd[i].revents)
 	    timeout = 0;
 	}

compat/poll/poll.h

@@ -59,6 +59,22 @@ typedef unsigned long nfds_t;
 
 extern int poll (struct pollfd *pfd, nfds_t nfd, int timeout);
 
+#if (defined _WIN32 || defined __WIN32__) && ! defined __CYGWIN__
+/*
+ * This poll() is emulated with MsgWaitForMultipleObjects(), which waits on at
+ * most MAXIMUM_WAIT_OBJECTS (64) objects. Two of those are never available for
+ * polled descriptors: poll() waits on its own event object, and QS_ALLINPUT
+ * adds the thread message queue. Sockets do not count, because they are all
+ * multiplexed onto that one event object; every other descriptor takes a wait
+ * slot of its own.
+ *
+ * Callers that poll one or more descriptors per child must keep the number of
+ * simultaneously live descriptors within this limit. Exceeding it fails with
+ * EINVAL.
+ */
+#define POLL_MAX_DESCRIPTORS 62
+#endif
+
 /* Define INFTIM only if doing so conforms to POSIX.  */
 #if !defined (_POSIX_C_SOURCE) && !defined (_XOPEN_SOURCE)
 #define INFTIM (-1)

compat/posix.h

@@ -133,6 +133,16 @@
 /* Pull the compat stuff */
 #include <poll.h>
 #endif
+
+/*
+ * compat/poll defines POLL_MAX_DESCRIPTORS to the largest number of
+ * descriptors its poll() emulation can wait on. A native poll() has no such
+ * limit, so callers that fan out one descriptor per child can clamp against
+ * this unconditionally.
+ */
+#ifndef POLL_MAX_DESCRIPTORS
+#define POLL_MAX_DESCRIPTORS INT_MAX
+#endif
 #ifdef HAVE_BSD_SYSCTL
 #include <sys/sysctl.h>
 #endif

hook.c

@@ -798,6 +798,7 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 
 		.processes = jobs,
 		.ungroup = jobs == 1,
+		.no_stdin_pipe = !options->feed_pipe,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,

parallel-checkout.c

@@ -671,6 +671,13 @@ int run_parallel_checkout(struct checkout *state, int num_workers, int threshold
 	if (parallel_checkout.nr < num_workers)
 		num_workers = parallel_checkout.nr;
 
+	/*
+	 * gather_results_from_workers() polls one pipe per worker, so the
+	 * worker count must stay within what poll() can wait on.
+	 */
+	if (num_workers > POLL_MAX_DESCRIPTORS)
+		num_workers = POLL_MAX_DESCRIPTORS;
+
 	if (num_workers <= 1 || parallel_checkout.nr < threshold) {
 		write_items_sequentially(state);
 	} else {

run-command.c

@@ -1658,6 +1658,9 @@ static int pp_start_one(struct parallel_processes *pp,
 		}
 		return 1;
 	}
+	if (opts->no_stdin_pipe && pp->children[i].process.in < 0)
+		BUG("get_next_task requested a stdin pipe despite "
+		    "no_stdin_pipe");
 	if (!opts->ungroup) {
 		pp->children[i].process.err = -1;
 		pp->children[i].process.stdout_to_stderr = 1;
@@ -1893,6 +1896,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	int i, code;
 	int timeout = 100;
 	int spawn_cap = 4;
+	size_t max_live;
 	struct parallel_processes_for_signal pp_sig;
 	struct parallel_processes pp = {
 		.buffered_output = STRBUF_INIT,
@@ -1902,6 +1906,20 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	const char *tr2_label = opts->tr2_label;
 	const int do_trace2 = tr2_category && tr2_label;
 
+	/*
+	 * Unless the caller handles its own output, pp_buffer_io() polls one
+	 * output pipe per child and, unless excluded by no_stdin_pipe, may also
+	 * poll an input pipe. Limit the number of live children so that all of
+	 * their descriptors fit in one poll() call.
+	 */
+	max_live = opts->processes;
+	if (!opts->ungroup) {
+		size_t fds_per_process = opts->no_stdin_pipe ? 1 : 2;
+
+		if (max_live > POLL_MAX_DESCRIPTORS / fds_per_process)
+			max_live = POLL_MAX_DESCRIPTORS / fds_per_process;
+	}
+
 	if (do_trace2)
 		trace2_region_enter_printf(tr2_category, tr2_label, NULL,
 					   "max:%"PRIuMAX,
@@ -1923,7 +1941,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)
 	while (1) {
 		for (i = 0;
 		    i < spawn_cap && !pp.shutdown &&
-		    pp.nr_processes < opts->processes;
+		    pp.nr_processes < max_live;
 		    i++) {
 			code = pp_start_one(&pp, opts);
 			if (!code)