feat: support Scatter with OpenMPI backend implementation

[GordonYang1]

Summary

This PR adds collective Scatter support to InfiniCCL with an OpenMPI-based backend implementation, along with an example program for functional verification and basic performance evaluation.

Scatter distributes one equally-sized block from a single root rank to every rank (the inverse of Gather). The public API is exposed through infinicclScatter() with the signature (sendbuff, recvbuff, count, datatype, root, comm, stream), consistent with the existing rooted collectives. The current implementation host-stages device buffers and delegates to a blocking MPI_Scatter internally.

Changes

• Public Scatter API

  • add the public API declaration for infinicclScatter() in include/comm.h;
  • the collective is exposed through the common communicator interface; the extern "C" entry is produced by the existing auto-generated bridge, so no manual wiring is needed.

• Base Scatter Wrapper

  • add src/base/scatter.h;
  • validate the communicator handle, datatype value, root rank range, and buffer pointers (the receive buffer is required on every rank, while the send buffer is only required on root) before dispatching to the backend implementation;
  • treat a zero count as a successful no-op;
  • return infinicclInvalidArgument for invalid user inputs.

• OpenMPI-based Scatter Implementation

  • add src/ompi/impl/scatter.h;
  • implement the collective with a blocking MPI_Scatter, allocating the host send buffer only on root, which holds the per-rank blocks to be distributed;
  • use temporary host-staging buffers for device-memory transfer (device-to-host of the full send buffer on root before, host-to-device of the received block on every rank after);
  • map DataType to the corresponding MPI_Datatype, and reject a count exceeding INT_MAX.

• Scatter Example

  • add examples/scatter.cc;
  • align the example structure and output style with the existing example programs such as all_reduce, all_gather, reduce_scatter, broadcast, and all_to_all;
  • root fills the block destined for rank r with (r + 1), and each rank verifies that it received its own (rank + 1) block;
  • report basic timing and bandwidth metrics in the same style as the other examples.

Platform and Backend Affected

Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU

Backend

• OpenMPI
• MPICH

Performance Impact

• No performance impact
• Performance improved
• Performance regression possible

This PR adds a new collective and does not change the performance of existing operations. For reference, the heterogeneous run (16 ranks, 4 MB per rank) measured Scatter at 61.420 ms, 1.91 GB/s bus bandwidth, 1.02 GB/s algorithm bandwidth.

Known Issues & Future Work

• The current OpenMPI Scatter implementation is blocking and does not overlap communication with computation. Future work may introduce stream-aware asynchronous execution.
• The current implementation allocates temporary host-staging buffers using malloc/free on every invocation. Future work may add reusable buffer pools, allocator caching, and pinned host memory to reduce per-call overhead.
• The current implementation rejects a count larger than INT_MAX instead of splitting it into INT_MAX-bounded chunks (as the point-to-point path does). Future work may add chunking for very large transfers.
• The current implementation performs GPU-to-Host and Host-to-GPU copies. While functionally correct, this is not optimal for GPU-intensive workloads. Future work may implement zero-copy GPU-GPU transfers or native GPU collectives (e.g. NCCL / CNCL) where supported.
• The current example intentionally follows the lightweight style of the existing example programs. Future dedicated tests may add broader coverage such as a non-zero root, zero-count calls, mismatched-buffer validation, and large-count stress cases.

Test Results

Validated on a MetaX–NVIDIA heterogeneous cluster over the OpenMPI backend via scripts/run_examples.py:

• server: NVIDIA, 8 GPUs, ranks 0–7 (built with Devices [cpu, nvidia], Backends [ompi]).
• test: MetaX, 8 GPUs, ranks 8–15 (built with Devices [cpu, metax], Backends [ompi]).
• 16 ranks total; message size 1,048,576 float32 (4 MB) per rank; 2 warm-up + 20 profiled iterations.

Test Involved Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU

Test Involved Backend

• OpenMPI
• MPICH

all_gather.log
all_reduce.log
all_to_all.log
broadcast.log
gather.log
reduce.log
reduce_scatter.log
scatter.log
send_recv.log

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from master — the branch is rebased cleanly on top of the current master.
• No fixup! / squash! / wip commits remain.

Scope and Design

• Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

• The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
• Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
• No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++).
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• N/A- Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
• Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
• Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

• N/A- Code is PEP 8 compliant; ruff check passes cleanly on CI (see .github/workflows/ruff.yml).
• N/A- ruff format --check passes cleanly — if not, run ruff format and commit the result.
• N/A- Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• N/A- Framework-specific conventions (e.g. lowercase pytest.skip messages without terminal period) are honored where applicable (CONTRIBUTING.md §Python).
• N/A- No blank line between the function signature and the body when there is no docstring or comment (CONTRIBUTING.md §Python).
• N/A- A blank line is present before and after if, for, and similar control-flow statements (CONTRIBUTING.md §Python).
• N/A- A blank line appears before each return, except when it directly follows a control-flow statement (CONTRIBUTING.md §Python).
• N/A- Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
• N/A- Type hints are added / kept consistent with the surrounding code.

Testing

• All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup.

Build, CI, and Tooling

• N/A- New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable.
• Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

• N/A- README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• N/A- Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• N/A- Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.