fix(infer): complete streamed requests independently within batches

[taking-lying-flat]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

When one request finishes before other requests in a Transformers streaming batch, its final event waits for the entire batch: _infer_stream() removes PAD tokens before checking whether the last token is PAD. Additionally, the worker forwards missing chunks as None, which the async consumer interprets as end-of-stream and can close a request before its final event arrives.

Detect completion from generated EOS IDs and template stop words before cleaning token IDs. Keep independent stop-word state for the consumer because generation runs ahead in another thread. Exclude the initial streamed prompt from completion detection and token counts, and preserve a terminating EOS when it shares its ID with PAD. A normal PAD does not end a request. Reuse existing batch-end handling for length-limit completion.

Skip missing chunks in the worker and close each async stream after its final response. Existing per-request completion flags prevent duplicate final events and tool-call parsing. Production changes are confined to transformers_engine.py.

Fix existing worker-test cleanup to stop its background loop before closing it and handle intentional test-worker shutdown cleanly. The PR modifies only existing files.

Experiment results

• python -m pytest tests/infer/test_prompt_usage.py tests/infer/test_transformers_worker.py -q: 11 tests and 20 subtests passed, with no deselected tests.
• Additional local regression checks covered early completion within a batch, EOS lists, shared EOS/PAD, first-token EOS, stop words and token sequences, buffered text, length limits, usage/logprobs, and prompt tokens. A paused producer verified that the completed request receives exactly one final event and closes independently.
• Eight additional cases passed through the real Hugging Face generate() implementation using a small randomly initialized GPT-2 model and deterministic logits on CPU.
• Local regression checks reproduce failures against the original streaming methods.
• All applicable pre-commit hooks and git diff --check passed.

No deployed-service latency benchmark was run.