[damccorm]

Note: Stacked on #40822 (MultiProcessShared spawn_process fixes). Once #40822 merges into master, only the RunInference commit (apache_beam/ml/inference/base.py and base_test.py) remains in this PR.

Summary

Addresses #40673 using approach (b) from the issue discussion: spawn shared MultiProcessShared models in dedicated processes (spawn_process=True).

Root Cause

When MultiProcessShared(..., always_proxy=True, spawn_process=False) is used, the process that initializes the shared object both runs the _SingletonRegistrar server (Server.accepter) on a background thread and connects to its own server via BaseProxy. If an unreachable BaseProxy is part of a reference cycle and Python's cyclic GC runs during an allocation on the accepter thread, BaseProxy._decref opens a blocking client connection to that same server while accepter is blocked inside _decref, deadlocking the server (and any callers waiting on _process_local_lock).

Changes

• apache_beam/ml/inference/base.py:
  • Pass spawn_process=True on the always_proxy=True MultiProcessShared handles (_RunInferenceDoFn._load_model, _RunInferenceDoFn._run_inference, load_model_status, _SharedModelWrapper, and finish_bundle's unsafe_hard_delete).
  • Use always_proxy=False for worker-local ModelManager so its process_wide=True metrics stay in the SDK worker without self-proxying.
• Tests (apache_beam/ml/inference/base_test.py):
  • Dedicated OS PID verification for single shared models and model_copies > 1.
  • End-to-end regression test reproducing cyclic-GC-on-accepter with reference-cycled model handles, local closures, and side-input model updates ([Bug]: Using MultiProcessShared handle can deadlock when always_proxy=True #40673).
  • Environment variable propagation to spawned model processes.