Fix: Rolling KV cache and top-k logit trimming (fixes #675)

[medmomoait]

Fixes #675

Changes

1. ChatSampler — Rolling KV Cache (ring buffer)

Adds rolling_cache and rolling_cache_preserve_tokens options to
ChatSampler. When enabled, old tokens are evicted from the KV cache
in a ring-buffer fashion before each turn, preventing context-exhaustion
OOM in long multi-turn conversations. A configurable prefix (e.g. system
prompt) can be protected from eviction.

2. SamplerLoop — Top-k logit trimming

Adds a top_k_logits option to SamplerLoop. When set, logits are
masked to top-k immediately after the forward pass, reducing the
transient VRAM footprint during sampling without changing the sampling
distribution.

Both changes default to the previous behavior (disabled), so this is
backward compatible.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.