[None][fix] Add KV cache V2 recompute pause path - #15252
Conversation
|
/bot run |
|
PR_Github #53824 [ run ] triggered by Bot. Commit: |
|
PR_Github #53824 [ run ] completed with state
|
8975655 to
e4db5f8
Compare
|
/bot run --disable-fail-fast |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (11)
🚧 Files skipped from review as they are similar to previous changes (11)
WalkthroughChangesRecompute-Pause Request Path
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant KVCacheV2Scheduler
participant SchedulerOutput
participant PyExecutor
participant LlmRequest
KVCacheV2Scheduler->>SchedulerOutput: add recompute_paused_requests
SchedulerOutput->>PyExecutor: propagate scheduled request IDs
PyExecutor->>PyExecutor: defer PP-synchronized teardown
PyExecutor->>LlmRequest: reset_for_recompute with unbounded input cap
PyExecutor->>PyExecutor: free KV resources and update paused statistics
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
tensorrt_llm/_torch/pyexecutor/resource_manager.py (1)
2563-2599:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy liftMake the host-tier fallback collective across ranks.
This fallback only disables the host tier on the rank that sees
KVCacheManagerPy(config)fail. Inworld_size > 1, another rank can keep the host tier while this rank retries GPU-only, leaving ranks with different cache-tier configs andhas_host_cache_tiervalues; Line 2512 already notes scheduling depends on synchronized per-rank capacity. Please allreduce a local “host init failed” flag and, if any rank fails, rebuild GPU-only on every rank, tearing down any successfully created host-tier manager first.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/resource_manager.py` around lines 2563 - 2599, The host-tier fallback mechanism in the __init__ method only disables the host tier on the rank experiencing the failure, causing inconsistent cache-tier configurations across ranks when world_size > 1. After the try-except block that catches CuError and KVCacheOutOfMemoryError during KVCacheManagerPy instantiation, add an allreduce operation to synchronize a local "host init failed" flag across all ranks. If the allreduce indicates any rank failed to initialize with the host tier, all ranks must rebuild their cache configuration and KVCacheManagerPy instance using GPU-only cache tiers. For ranks that already successfully created the host-tier KVCacheManagerPy instance in self.impl before the allreduce, ensure that instance is properly torn down before rebuilding with the GPU-only configuration. Update both self.has_host_cache_tier and self.kv_cache_manager_py_config consistently across all ranks.tensorrt_llm/_torch/pyexecutor/py_executor.py (2)
2871-2877:⚠️ Potential issue | 🟠 Major | ⚡ Quick winRun recompute-pause cleanup before retrying the loop.
If
_check_benchmark_disagg_gate()returnsshould_retry, both loopscontinuebefore the new recompute terminate/pause helpers run. Any non-emptyscheduled_batch.recompute_paused_requestsfrom scheduling is left without the executor-side reset, so the next schedule can see a request whose scheduler-side KV state was already suspended.Proposed fix
can_forward, should_retry = self._check_benchmark_disagg_gate( scheduled_batch, can_forward) if should_retry: + self._terminate_recompute_paused_requests(scheduled_batch) + self._pause_recompute_paused_requests(scheduled_batch) continue self._terminate_recompute_paused_requests(scheduled_batch) self._pause_recompute_paused_requests(scheduled_batch) @@ can_forward, should_retry = self._check_benchmark_disagg_gate( scheduled_batch, can_forward) if should_retry: + self._terminate_recompute_paused_requests(scheduled_batch) + self._pause_recompute_paused_requests(scheduled_batch) continue self._terminate_recompute_paused_requests(scheduled_batch)Also applies to: 3259-3264, 3412-3414
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/py_executor.py` around lines 2871 - 2877, The recompute cleanup calls to _terminate_recompute_paused_requests() and _pause_recompute_paused_requests() are positioned after the should_retry check, so they are skipped when _check_benchmark_disagg_gate() returns should_retry=True and the loop continues. This leaves scheduled_batch.recompute_paused_requests without executor-side reset on retry. Move both _terminate_recompute_paused_requests() and _pause_recompute_paused_requests() calls to execute before the if should_retry: continue block at all three affected locations: tensorrt_llm/_torch/pyexecutor/py_executor.py lines 2871-2877 (anchor), 3259-3264 (sibling), and 3412-3414 (sibling), so that the cleanup always happens regardless of whether a retry occurs.
1343-1358:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winSnapshot paused KV tokens before overlap can mutate recompute-paused requests.
_update_iter_stats()may consume a storedScheduledRequestsafter the next overlap scheduling pass has resumed or mutated the same recompute-paused request, so the livereq.get_num_tokens(0)at Line 1663 can report the wrong iteration. Capturenum_paused_kv_tokensinScheduledBatchStatsalongside the paused count and prefer that snapshot here.Proposed fix
`@dataclasses.dataclass` class ScheduledBatchStats: @@ num_gen_requests: Optional[int] = None num_gen_kv_tokens: Optional[int] = None num_paused_requests: Optional[int] = None + num_paused_kv_tokens: Optional[int] = None @@ num_paused_requests = 0 + num_paused_kv_tokens = 0 paused_requests = (scheduled_batch.paused_requests + scheduled_batch.recompute_paused_requests) for req in paused_requests: if filter_dummies and self._is_stats_dummy_request(req): continue num_paused_requests += 1 + try: + num_paused_kv_tokens += req.get_num_tokens(0) + except RuntimeError: + pass return ScheduledBatchStats( @@ num_gen_requests=num_gen_requests, num_gen_kv_tokens=num_gen_kv_tokens, num_paused_requests=num_paused_requests, + num_paused_kv_tokens=num_paused_kv_tokens, ) @@ # Total KV context length summed across paused (preempted-decode) # requests — were decoding but got evicted back to the waiting # pool for this iteration. - num_paused_kv_tokens = 0 - for req in paused_requests: - if self._is_stats_dummy_request(req): - continue - try: - num_paused_kv_tokens += req.get_num_tokens(0) - except RuntimeError: - pass + if scheduled_batch_stats.num_paused_kv_tokens is not None: + num_paused_kv_tokens = int( + scheduled_batch_stats.num_paused_kv_tokens) + else: + num_paused_kv_tokens = 0 + for req in paused_requests: + if self._is_stats_dummy_request(req): + continue + try: + num_paused_kv_tokens += req.get_num_tokens(0) + except RuntimeError: + passAlso applies to: 1655-1665
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/py_executor.py` around lines 1343 - 1358, In the ScheduledBatchStats return statement at the end of this method, add calculation and capture of num_paused_kv_tokens alongside num_paused_requests. While iterating through paused_requests to count num_paused_requests, also sum up the token counts (using req.get_num_tokens(0)) for each non-dummy request to calculate num_paused_kv_tokens. Then add the num_paused_kv_tokens field to the ScheduledBatchStats constructor call alongside the other token count fields. This creates a snapshot of paused KV tokens at the time of batch stats creation, which can then be used elsewhere instead of accessing potentially mutated request state at a later time.
🧹 Nitpick comments (2)
tensorrt_llm/_torch/pyexecutor/py_executor.py (1)
5122-5158: ⚡ Quick winAnnotate the new recompute-pause helpers.
The new helper methods omit argument/return annotations; add
LlmRequest/ScheduledRequestsand-> Noneso the new private contract remains type-checkable.Proposed fix
- def _pause_recompute_request(self, req): + def _pause_recompute_request(self, req: LlmRequest) -> None: @@ def _terminate_recompute_paused_requests( - self, scheduled_batch: ScheduledRequests): + self, scheduled_batch: ScheduledRequests) -> None: @@ def _pause_recompute_paused_requests(self, - scheduled_batch: ScheduledRequests): + scheduled_batch: ScheduledRequests) -> None:As per coding guidelines, “Always annotate functions. Make the return type
Noneif the function does not return anything.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/py_executor.py` around lines 5122 - 5158, Add complete type annotations to the three recompute-pause helper methods to comply with coding guidelines. For the `_pause_recompute_request` method, add the `req` parameter type annotation as `LlmRequest`. For all three methods `_pause_recompute_request`, `_terminate_recompute_paused_requests`, and `_pause_recompute_paused_requests`, add the return type annotation `-> None` since they do not return any values. The `scheduled_batch` parameter in the latter two methods is already annotated with `ScheduledRequests`, but the return type annotation is missing from all three methods.Source: Coding guidelines
tests/unittest/pyexecutor/test_iter_stats_populate.py (1)
406-420: ⚡ Quick winAdd a recompute-paused dummy-filter regression case.
This test validates paused-count/KV aggregation, but it does not yet guard the dummy-filter behavior for
recompute_paused_requests. Please add a focused case (or extendtest_dummy_filtering_on_kv_token_fields) that mixes dummy and non-dummy recompute-paused requests and asserts only real work contributes to paused KV metrics.Based on learnings from the PR objectives, recompute-paused request handling is a primary change surface and should be explicitly covered in stats tests.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/pyexecutor/test_iter_stats_populate.py` around lines 406 - 420, Add a regression test case to validate that the dummy-filter behavior correctly handles recompute_paused_requests. Either create a new focused test or extend the existing test_dummy_filtering_on_kv_token_fields function to mix dummy and non-dummy recompute-paused requests using _StubRequest objects with appropriate dummy-filtering flags, then assert that only the real work (non-dummy requests) contributes to the paused KV token metrics in the returned stats object. This ensures the recompute-paused request handling, which is a primary change surface, is explicitly covered by stats validation tests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py`:
- Around line 67-106: The SchedulerOutput namedtuple now contains six fields
instead of three, but existing code unpacks the scheduler output into only three
variables (e.g., fitting, disagg, paused = scheduler.schedule_request(...)),
which will raise a ValueError at runtime. Update all unpacking statements that
destructure SchedulerOutput to account for all six fields by either using
attribute access (output.context_requests, output.generation_requests, etc.) or
explicitly unpacking all six values including the new recompute_paused_requests
field. Pay special attention to test code patterns where this unpacking occurs
frequently.
---
Outside diff comments:
In `@tensorrt_llm/_torch/pyexecutor/py_executor.py`:
- Around line 2871-2877: The recompute cleanup calls to
_terminate_recompute_paused_requests() and _pause_recompute_paused_requests()
are positioned after the should_retry check, so they are skipped when
_check_benchmark_disagg_gate() returns should_retry=True and the loop continues.
This leaves scheduled_batch.recompute_paused_requests without executor-side
reset on retry. Move both _terminate_recompute_paused_requests() and
_pause_recompute_paused_requests() calls to execute before the if should_retry:
continue block at all three affected locations:
tensorrt_llm/_torch/pyexecutor/py_executor.py lines 2871-2877 (anchor),
3259-3264 (sibling), and 3412-3414 (sibling), so that the cleanup always happens
regardless of whether a retry occurs.
- Around line 1343-1358: In the ScheduledBatchStats return statement at the end
of this method, add calculation and capture of num_paused_kv_tokens alongside
num_paused_requests. While iterating through paused_requests to count
num_paused_requests, also sum up the token counts (using req.get_num_tokens(0))
for each non-dummy request to calculate num_paused_kv_tokens. Then add the
num_paused_kv_tokens field to the ScheduledBatchStats constructor call alongside
the other token count fields. This creates a snapshot of paused KV tokens at the
time of batch stats creation, which can then be used elsewhere instead of
accessing potentially mutated request state at a later time.
In `@tensorrt_llm/_torch/pyexecutor/resource_manager.py`:
- Around line 2563-2599: The host-tier fallback mechanism in the __init__ method
only disables the host tier on the rank experiencing the failure, causing
inconsistent cache-tier configurations across ranks when world_size > 1. After
the try-except block that catches CuError and KVCacheOutOfMemoryError during
KVCacheManagerPy instantiation, add an allreduce operation to synchronize a
local "host init failed" flag across all ranks. If the allreduce indicates any
rank failed to initialize with the host tier, all ranks must rebuild their cache
configuration and KVCacheManagerPy instance using GPU-only cache tiers. For
ranks that already successfully created the host-tier KVCacheManagerPy instance
in self.impl before the allreduce, ensure that instance is properly torn down
before rebuilding with the GPU-only configuration. Update both
self.has_host_cache_tier and self.kv_cache_manager_py_config consistently across
all ranks.
---
Nitpick comments:
In `@tensorrt_llm/_torch/pyexecutor/py_executor.py`:
- Around line 5122-5158: Add complete type annotations to the three
recompute-pause helper methods to comply with coding guidelines. For the
`_pause_recompute_request` method, add the `req` parameter type annotation as
`LlmRequest`. For all three methods `_pause_recompute_request`,
`_terminate_recompute_paused_requests`, and `_pause_recompute_paused_requests`,
add the return type annotation `-> None` since they do not return any values.
The `scheduled_batch` parameter in the latter two methods is already annotated
with `ScheduledRequests`, but the return type annotation is missing from all
three methods.
In `@tests/unittest/pyexecutor/test_iter_stats_populate.py`:
- Around line 406-420: Add a regression test case to validate that the
dummy-filter behavior correctly handles recompute_paused_requests. Either create
a new focused test or extend the existing
test_dummy_filtering_on_kv_token_fields function to mix dummy and non-dummy
recompute-paused requests using _StubRequest objects with appropriate
dummy-filtering flags, then assert that only the real work (non-dummy requests)
contributes to the paused KV token metrics in the returned stats object. This
ensures the recompute-paused request handling, which is a primary change
surface, is explicitly covered by stats validation tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: de6769ea-1d1f-44d3-81fd-d410cf65d726
📒 Files selected for processing (9)
tensorrt_llm/_torch/pyexecutor/py_executor.pytensorrt_llm/_torch/pyexecutor/resource_manager.pytensorrt_llm/_torch/pyexecutor/scheduler/scheduler.pytensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.pytests/unittest/_torch/executor/test_kv_cache_v2_scheduler.pytests/unittest/_torch/executor/test_py_scheduler.pytests/unittest/_torch/executor/test_scheduler_serializable_output.pytests/unittest/pyexecutor/test_iter_stats_populate.pytests/unittest/pyexecutor/test_recompute_pause.py
|
PR_Github #54461 [ run ] triggered by Bot. Commit: |
|
/bot run |
|
PR_Github #54543 [ run ] triggered by Bot. Commit: |
|
PR_Github #54461 [ run ] completed with state |
|
PR_Github #54543 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #54597 [ run ] triggered by Bot. Commit: |
|
PR_Github #54597 [ run ] completed with state
|
a541ad1 to
1e1d7a0
Compare
|
/bot run |
|
/bot run --disable-fail-fast |
|
PR_Github #54826 [ run ] triggered by Bot. Commit: |
|
PR_Github #54827 [ run ] triggered by Bot. Commit: |
|
[by Codex] @lowsfer Friendly reminder to review this PR as the primary KV-cache-manager reviewer when you have a chance. Thanks! |
|
[by Codex] @lowsfer Could you please review this PR? Thank you! |
|
[by Codex] @lowsfer Friendly review reminder: this PR is awaiting your review. Thanks! |
20a34d1 to
5a57937
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #63416 [ run ] triggered by Bot. Commit: |
|
PR_Github #63416 [ run ] completed with state
|
|
|
||
| if self._is_kv_manager_v2: | ||
| self._terminate_recompute_paused_requests(scheduled_batch) | ||
| self._pause_recompute_paused_requests(scheduled_batch) |
There was a problem hiding this comment.
Nice cleanup of the V2 pause path. One concern about this call landing in the PP loop specifically: this is the only loop where _disagg_pp_termination_handler is live (constructed at :961 for pp_size > 1 + disagg), and there _terminate_request (:6711) only stashes the request — the actual free waits for a ring consensus. The pause on the next line runs immediately, and pause() ends in mSeqSlot.reset(), so the victim becomes re-schedulable while its old slot mapping is still present and add_slot (resource_manager.py:2333) asserts; if the consensus lands first, it frees a request that has meanwhile re-acquired a slot and KV. Could the recompute victims bypass the deferred handler here, or the pause wait until termination has actually landed?
There was a problem hiding this comment.
Added a buffer list to defer the termination of the requests
|
/bot run --disable-fail-fast |
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
PR_Github #63981 [ run ] triggered by Bot. Commit: |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py (1)
94-103: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd annotations to new helper functions.
The changed helper functions omit required parameter or return annotations.
tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py#L94-L103: add theSchedulerOutputreturn annotation to__new__.tensorrt_llm/_torch/pyexecutor/py_executor.py#L6727-L6770: annotate termination-helper parameters andNonereturns.tensorrt_llm/_torch/pyexecutor/py_executor.py#L7141-L7178: annotate recompute-pause helper parameters andNonereturns.As per coding guidelines, “Annotate every function.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py` around lines 94 - 103, Annotate every changed helper function: in tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py lines 94-103, add SchedulerOutput as the return annotation for Scheduler.__new__; in tensorrt_llm/_torch/pyexecutor/py_executor.py lines 6727-6770, annotate the termination-helper parameters and None return; and in lines 7141-7178, annotate the recompute-pause helper parameters and None return.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py`:
- Around line 94-103: Annotate every changed helper function: in
tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py lines 94-103, add
SchedulerOutput as the return annotation for Scheduler.__new__; in
tensorrt_llm/_torch/pyexecutor/py_executor.py lines 6727-6770, annotate the
termination-helper parameters and None return; and in lines 7141-7178, annotate
the recompute-pause helper parameters and None return.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 77b51e64-160b-4d63-b91e-5995f29e40a9
📒 Files selected for processing (11)
tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.pytensorrt_llm/_torch/pyexecutor/llm_request.pytensorrt_llm/_torch/pyexecutor/py_executor.pytensorrt_llm/_torch/pyexecutor/sampler/finish_reasons.pytensorrt_llm/_torch/pyexecutor/scheduler/scheduler.pytensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.pytests/unittest/_torch/executor/test_iter_stats_populate.pytests/unittest/_torch/executor/test_kv_cache_v2_scheduler.pytests/unittest/_torch/executor/test_py_scheduler.pytests/unittest/_torch/executor/test_recompute_pause.pytests/unittest/_torch/executor/test_scheduler_serializable_output.py
🚧 Files skipped from review as they are similar to previous changes (4)
- tests/unittest/_torch/executor/test_scheduler_serializable_output.py
- tests/unittest/_torch/executor/test_py_scheduler.py
- tests/unittest/_torch/executor/test_kv_cache_v2_scheduler.py
- tensorrt_llm/_torch/pyexecutor/scheduler/scheduler_v2.py
|
PR_Github #63981 [ run ] completed with state
|
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
Signed-off-by: Yi Zhang <187001205+yizhang-nv@users.noreply.github.com>
15802a1 to
cee6bbc
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
/bot run |
|
PR_Github #64242 [ run ] triggered by Bot. Commit: |
|
PR_Github #64242 [ run ] completed with state
|
6 participants
Dev Engineer Review
LlmRequeststate initialization for recompute resets.SchedulerOutputconstruction with an empty default list.QA Engineer Review
test_recompute_pause_does_not_apply_executor_max_input_lentest_recompute_pause_defers_reset_until_pp_consensustest_terminal_request_does_not_recompute_after_pp_consensustest_paused_decode_requeststest_serializable_scheduler_output_round_triptest_full_pipeline_output_structuretests/integration/test_lists/were changed.test-db/andqa/is unavailable.Description
Cherry-pick/adapt of #14816 onto
main.This adds KV cache V2 handling for recompute-paused requests in the PyTorch executor path:
maindisk cache tier logic while adding host cache tier detection and GPU-only fallback when host tier setup failsOpened as draft so the main-branch adaptation can be reviewed before marking ready.
Test Coverage
python3 -m pytest -s tests/unittest/_torch/executor/test_kv_cache_v2_scheduler.py tests/unittest/_torch/executor/test_scheduler_serializable_output.py(168 passed)python3 -m pytest -s tests/unittest/pyexecutor/test_iter_stats_populate.py tests/unittest/_torch/executor/test_py_scheduler.py -k "paused or full_pipeline_output_structure"(3 passed, 150 deselected)python3 -m pytest -s tests/unittest/pyexecutor/test_iter_stats_populate.py tests/unittest/pyexecutor/test_recompute_pause.py(25 passed)PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.