Skip to content

feat(server): execution guarantees, async lifecycle, and strict wire contracts - #286

Open
Yunnglin wants to merge 33 commits into
mainfrom
feat/server-execution-guarantees
Open

feat(server): execution guarantees, async lifecycle, and strict wire contracts#286
Yunnglin wants to merge 33 commits into
mainfrom
feat/server-execution-guarantees

Conversation

@Yunnglin

@Yunnglin Yunnglin commented Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Hardens Twinkle's serving path with explicit execution bounds, an asynchronous submit/retrieve lifecycle, and strict client/server wire contracts. The client and server now share protocol models, reject invalid requests before GPU work is enqueued, and preserve complete failures through ErrorPayload.

Key changes

Execution guarantees

  • Applies effective time bounds to synchronous dispatch paths and moves blocking backend work behind call_backend plus a per-replica admission gate.
  • Removes silent degradation and the TWINKLE_FAIL_FAST switch; post-timeout liveness probing records backend health.
  • Adds total and per-token timeouts to sampler streams; failures surface as ErrorPayload frames.

Async request lifecycle

  • Adds the run_submit / submit_and_peek submit shell and persistent FutureRecord state.
  • Twinkle-native endpoints return TaskEnvelope; the client future layer retrieves terminal results without holding the original HTTP connection open.
  • Deduplicates retried gradient-mutating operations by (session, adapter, seq_id).

Shared strict wire contracts

  • Moves shared request and response models to twinkle.protocol.types; the client builds typed requests and serializes them once before transport.
  • Enforces StrictRequest at Twinkle-native HTTP ingress: unknown top-level fields, body tokens, incompatible backend fields, and unavailable endpoints fail before enqueueing work.
  • Validates inline input batches for strict integer leaves, homogeneous shape, and extension-field round trips.
  • Separates model- and sampler-domain request names to prevent schema collisions, including sampler dict LoRA configuration handling.

Client and structural cleanup

  • Standardizes ClientTransport / ClientContext ownership for routing, session, auth, retry, and request serialization.
  • Enables client linting, normalizes affected client code, and completes server/client module renames and auto-agent/rollout refactors.
  • Removes file persistence mode; supported persistence modes are memory and redis.

Breaking changes

  • Twinkle-native control-plane and inline-data requests are strict: unknown top-level fields, body-supplied tokens, bool/float token IDs, and heterogeneous inline batches are rejected.
  • Client and server must be upgraded together; no compatibility or lax parsing mode is provided.
  • File persistence configuration is removed; deployments must use memory or redis.

Validation

  • pre-commit passes for all touched files.
  • tinker_myl targeted validation: 40 passed, 3 skipped, covering request construction, async client component wire shapes, TensorDict import behavior, and server request-wire contracts.

@Yunnglin
test(contract): refresh client_api_baseline before execution-guarante…
a98c5f0 
…es work

T0.1: capture current HEAD client-facing surface so the post-refactor
comparison (T8.1) shows only this spec's changes. No code changes.
@Yunnglin
feat(infra): establish Sync_Dispatch_Path time bound (Phase 1, T1.1-T…
c359789 
…1.6)

- execute_all_sync forwards timeout to ray.get (T1.1)
- resolve effective ray.get timeout before choosing execute_method and flip
  priority to 'decorator wins, instance is fallback' at both dispatch sites;
  fix the 0-treated-as-falsy trap; bound __len__/__next__ bare ray.get (T1.2, T1.5)
- decorator timeout=10 on ping; timeout=3600 on save/add_adapter_to_model/
  resume_from_checkpoint/tinker_load/load_full_weights_from_path (T1.3)
- TaskQueueConfig default execution_timeout 120->1800; effective_execution_timeout
  (0 -> 3600) as the single bound source; startup warning on 0 (T1.4)
- infra unit tests, no GPU/Megatron/server deps (T1.6)
@Yunnglin
feat(contract): ErrorPayload contract + type base classes (Phase 2, T…
1b030ea 
…2.1-T2.4, T6.1)

- twinkle_client/types/base.py: StrictRequest/ResponseModel/DataModel + backend_only()
  helper/reader; naming rulings in module docstring (defined, not applied) (T6.1)
- twinkle_client/types/errors.py: ErrorCategory + ErrorPayload(ResponseModel) (T2.1)
- task_errors.py: task_error_payload builds ErrorPayload dict (request_id/error_code,
  traceback split+tail-trim, User carries no traceback); error_payload_from_stored
  backfills legacy two-field payloads without ValidationError (T2.2)
- worker: single-line error summary + full traceback in traceback field; TimeoutError
  and Ray_Get_Timeout -> 504/Server, others -> 500/Server (T2.3)
- tests for ErrorPayload + updated task_errors test (T2.4)
@Yunnglin
feat(contract): QueueStateLiteral + naming disambiguation guard (Phas…
b6e0beb 
…e 6, T6.2-T6.3)

- QueueStateLiteral in types/errors.py, values sourced to match server QueueState;
  consistency test asserts equal value sets (T6.2)
- T6.3 realized as a guard test (per user ruling): twinkle_client already shares
  18 public names with tinker.types by design (tinker-compatible client), so the
  literal 'no intersection' cannot hold without renaming twinkle. Guard instead
  asserts no src/twinkle module binds a tinker and a twinkle_client type to the
  same local name (tinker must be aliased when both coexist).
@Yunnglin
feat(server): Blocking_Call_Boundary + Admission_Gate (Phase 3, T3.1-…
e46345b 
…T3.8)

- call_backend: dedicated ThreadPoolExecutor (no max_workers=1) + per-replica
  opt-in Admission_Gate; gate released from the worker thread's finally so a
  wait_for-cancelled coroutine cannot free it while the call is still in flight;
  fast-fail BackendBusyError when the gate is held by a leaked call (T3.1)
- ModelManagement enables the gate; SamplerManagement does not
- worker maps BackendBusyError -> 503/Server
- mechanical: 27 model/twinkle + 14 model/tinker + 8 sampler/twinkle + 4
  sampler/tinker + 3 model/app direct backend calls -> await call_backend (T3.2-T3.5)
- check_model_health async + admit=False ping; _cleanup_adapter via gate; /healthz
  awaits (T3.5, T3.6)
- AST static check over src/twinkle/server/** + shared exemptions file (T3.7)
- blocking-boundary integration tests (T3.8)
@Yunnglin
feat(server): post-timeout actor recovery + two time bounds (Phase 4,…
948daaa 
… T4.1-T4.3)

- set _ray_get_timeout = effective execution timeout on model/sampler backends,
  effective for both sync and async dispatch (T4.1)
- ComputeWorker fires an optional on_backend_timeout hook after a timeout;
  ModelManagement probes actor liveness (admit=False ping) and sets a health bit
  that /healthz reflects (503) and a successful probe auto-clears (T4.2)
- worker skips a dequeued task whose record is already terminal (R3#8); document
  record-terminal (queue_timeout+T) vs resource-release (Collect_Width*T) bounds in
  EN+ZH Server docs (T4.3)
@Yunnglin
feat(state): future record survival + ownership hygiene (Phase 5, T5.…
742faf7 
…1-T5.6)

- FutureRecord.replica_id set at creation, never overwritten (T5.1)
- ReplicaRegistry last_seen as a separate key (max_loras type unchanged);
  refreshed in _on_request_start; ModelManager.get_alive_replica_ids (T5.2)
- cleanup_expired rewrite: never deletes a non-terminal record; orphans and
  over-absolute-ttl records are written failed; signature adds alive_replica_ids
  + absolute_ttl; ServerState.set_execution_bounds injects queue_timeout/T/
  Collect_Width; age uses the stored-timestamp clock convention (T5.3)
- do-not-regress guard extended to terminal->terminal (warn on different,
  silent drop on same) (T5.4)
- sampler _stream_generator: 60s per-get + total-lifetime bound, ray Queue
  shutdown in finally, empty-actor structured error (T5.5)
- state hygiene tests via Ray-free FileBackend (T5.6)
@Yunnglin
refactor: remove all silent degradation mechanisms (Phase 7, T7.1-T7.9)
f461fc3 
- delete TWINKLE_FAIL_FAST from the 4 cookbook server configs (T7.1)
- delete Layer 1 (safe_loss/SafeLossWrapper/_zero_loss) + OptimizerGroup.__setattr__
  auto-wrap hook + the test_micro_batch safe_loss import/case (T7.2)
- delete Layer 2 (@nccl_safe decorator, _force_zero_backward, _iter_model_params)
  and its two decoration points + import in transformers_model (T7.3)
- delete Layer 4: the forward_step_func post-processing try/except in megatron.py,
  exceptions now propagate; drop the _is_fail_fast import (T7.4)
- delete Fail_Fast_Switch: _is_fail_fast + env_propagation NCCL_SAFE_ENV_KEYS/
  build_nccl_safe_env_vars and its call (T7.5)
- nccl_safe_megatron rewritten to unconditional rank-attributed log + re-raise,
  no tinker/forward_only params, no degraded return; module docstring records the
  lost coverage window (T7.6, T7.9)
- rewrite both GPU-gated e2e tests to assert failed-terminal + subsequent-success
  and drop degradation symbols (T7.7)
- unified static check asserting the 8 symbols are absent (T7.8)
@Yunnglin
test(contract): zero-wire-change guard + CI/pre-commit closeout (Phas…
141b812 
…e 8, T8.1/T8.2/T8.4)

- test_client_api_contract.py: OpenAPI surface == pre-impl baseline (T0.1),
  schedule_task_and_wait retained, only new client modules base.py/errors.py (T8.1)
- new checks are auto-run by CI 'pytest tests'; lint.yaml runs pre-commit --all-files;
  applied yapf/isort/pyupgrade auto-fixes so all hooks pass on changed src files (T8.2)
- tests/server/README.md documents the mock-backend evidence boundary (T8.4)
@Yunnglin
fix(state): store state timestamps as UTC so expiry is timezone-indep…
d54d9a1 
…endent

Root cause: _now_iso() wrote naive local time while _parse_timestamp() reads a
naive ISO string as UTC; compared against a time.time()-based cutoff this skewed
every expiry check by the host's UTC offset (premature deletion west of UTC,
over-retention east; invisible on UTC/CI hosts).

Fix (option A): _now_iso() now emits UTC-aware ISO; future_manager reuses it for
record writes and its cleanup 'now' reverts to time.time() (both UTC epoch).
_parse_timestamp is unchanged and still parses legacy naive records as UTC
(same as before). Not a wire-schema change (R8 unaffected).

Adds a timezone-independent regression test asserting a freshly written record
parses to within 1s of time.time(); the suite passes under TZ=America/Los_Angeles.
@Yunnglin
Merge branch 'main' into feat/server-execution-guarantees
250c380 
# Conflicts:
#	src/twinkle/server/model/app.py
@Yunnglin
fix: restore multi lora state after load failure
f978104
@Yunnglin
feat(server): finalize execution guarantees (backend boundary + strea…
26a4311 
…m/gen bounds)

Complete the server-execution-guarantees spec by routing the remaining
sampler paths through the Blocking_Call_Boundary and bounding every
long-lived operation:

- sampler: submit/collect/cancel generation, unload, and streaming now go
  through call_backend; _await_generation is bounded by the effective
  execution timeout and _stream_queue enforces a double timeout (total +
  per-token) instead of blocking indefinitely.
- task queue: thread collect_width through _init_task_queue so state
  hygiene computes the absolute survival TTL without a separate
  set_execution_bounds hop.
- errors: stream/generation failures surface as ErrorPayload wire frames
  rather than silent drops.
- deps: bump tinker to 0.29.0 (python>=3.11) and refresh poetry.lock.
- tests/docs: add error-wire, stream-guarantee and nccl_safe coverage;
  refresh the client API contract baseline (zero wire change).
@Yunnglin
fix: revert tinker to 0.16.1 and keep request types FastAPI-compatible
d56bcca 
tinker 0.29.0 turned its tensor request types into dataclasses, which
breaks FastAPI OpenAPI generation. Revert the pin and the code that had
been migrated to the 0.29.0 API:

- pyproject: tinker 0.29.0 -> 0.16.1; restore poetry.lock to match
- sampler/tinker_handlers: use 0.16.1 SampledSequence/SampleResponse fields
- contract harness: fastapi 0.136 ModelField compat; normalize :path route
  converters so the baseline stores client-facing paths
- regenerate client_api_baseline.json against tinker 0.16.1
- test fixtures: add call_backend/_task_queue_config; drop 0.29-only assertion
@Yunnglin
fix(server): harden tinker GRPO execution guarantees
4bc9beb
@Yunnglin
test(server): simplify execution guarantee coverage
fc07df5
@Yunnglin
feat(server): converge twinkle-native requests to submit/retrieve lif…
9ea3272 
…ecycle

Implements the server-request-lifecycle spec (Part 2). A single HTTP request's
server-side duration is now decoupled from task execution time: submit enqueues
and returns a TaskEnvelope immediately, and the client's future layer polls a
dedicated retrieve endpoint.

The motivation is narrow and load-bearing: the client's per-request timeout was
600s while most ingress gateways cut idle connections at 60s, so data-plane
endpoints (forward_backward, sample) were already unusable behind a real gateway.
Every single HTTP request is now bounded by the 30s long-poll window regardless of
how long the task runs. This does not improve throughput or training speed -- the
compute queue is still serial and GPU utilisation is unchanged.

Server:
- new twinkle/server/lifecycle/: envelope.py (the one FutureRecord -> TaskEnvelope
  mapping point), poll_config.py (single declaration of the long-poll window and
  interval, shared by both retrieve endpoints), submit.py (run_submit shell plus the
  to_backend_inputs / backend_kwargs / input_metrics seams left for Part 3)
- new POST /twinkle/retrieve_future and POST /twinkle/cancel
- preflight now raises RequestRejectedError subclasses, so a rejected request
  returns a real status code and writes zero future records
- TwinkleServerError handler puts ErrorPayload fields at the response top level
- delete schedule_task_and_wait, run_task (both copies), QueuedTask.completion,
  _complete_result/_complete_error and persist_status; the future record is now the
  only delivery channel for results and failures
- delete TaskStatus.RATE_LIMITED (limiting is now HTTP 429), the get_state_dict
  endpoint and the upload_status endpoint

Client:
- new types/lifecycle.py (TaskEnvelope), _future.py (the only polling loop),
  exceptions.py (TwinkleHTTPError / TaskFailedError / TaskCancelledError /
  TaskWaitTimeoutError / TaskRecordLostError)
- three separate 600s timeout literals collapse into _HTTP_TIMEOUT = 90
- public methods keep their synchronous signatures and return types, so cookbook
  scripts and integration tests are unchanged

Breaking changes: queued endpoints return TaskEnvelope instead of a business model;
task failure raises TaskFailedError (HTTP 200 + payload) instead of requests.HTTPError
(HTTP 500); get_state_dict is removed (use save + read the checkpoint).

Verified on real PPU hardware (Qwen3.5-4B, 8x ZW810): SFT/DPO/GRPO x twinkle/tinker
on the transformers backend and SFT x twinkle/tinker on megatron, 8/8 passing with
losses identical to the pre-refactor run. Unit suite: 360 passed, 0 failed.
@Yunnglin
refactor(server): extract sampler seams and rename session-resource m…
d848a5a 
…ixins

- move sampler weight resolution to sampler/weights.py and streaming
  bridge to sampler/backends/streaming.py so handlers stay thin
- rename utils/lifecycle to utils/session_resource: the package holds
  session-scoped resource mixins, not the request lifecycle owned by
  twinkle/server/lifecycle
- add static guards for adapter-name mapping and package-root imports
- declare grimp test dependency used by the import-boundary guards
@Yunnglin
feat(server): request schema — strict wire models, preflight, and nam…
b928a17 
…ing disambiguation

Move HTTP-boundary-decidable request problems out of the async training path:

- Single shared request models (twinkle_client.types) with field roles
  (control / backend_kwarg / passthrough); client builds via build_request +
  model_dump_json instead of hand-assembling json_data.
- StrictRequest on all twinkle-native routes; unified RequestValidationError ->
  ErrorPayload (422/501) registered on the shared deployment app builder.
- Wire schema for inline `inputs` validated at HTTP ingress (strict int leaves,
  homogeneous batch, extension-field preservation); single shared `is_encoded`
  predicate in twinkle.data_format.encoding replaces three copies.
- run_submit preflight (assert_request_supported) rejects backend-incompatible
  fields and unavailable endpoints before seq claim / enqueue -> zero DP ranks.
- Passthrough keys forwarded unjudged (no spelling heuristic).

Naming disambiguation (client/server contract fix):
- Prefix sampler-domain models (SamplerAddAdapterRequest / SetTemplate* /
  CreateResponse) so they no longer collide with model.py; the sampler handler
  now binds sampler_types explicitly. Fixes add_adapter_to_sampler validating
  against model.py's `config: Optional[str]` and rejecting the dict the client
  sends.
- Remove dead model.AddAdapterResponse and server.WeightsInfoResponse.
- training.py response envelopes inherit ResponseModel.
- Regenerate contract route inventory; add regression tests pinning the sampler
  binding and the dict-config acceptance.
@Yunnglin
style(server): satisfy pre-commit lint on request-schema
4f1addb 
Auto-fixes flagged by CI on b928a17 (pre-commit run --all-files):
- sampler/twinkle_handlers.py: isort import order + wrap the 122-char
  create() signature (E501).
- validation/backend_compat.py: pyupgrade Optional[str] -> str | None.
@Yunnglin
style(client): enforce lint on src/twinkle_client and normalize exist…
637c53d 
…ing code

Remove src/twinkle_client from the pre-commit exemptions (flake8/isort/yapf/
pyupgrade + the whitespace/EOL/quote fixers) so the client is linted like the
server, and bring the existing (previously-unchecked) client code to a green
`pre-commit run --all-files`:

- isort/yapf/pyupgrade/double-quote normalization across the client package.
- Wrap over-length tool/description/error strings (implicit concatenation,
  content preserved) in auto/agent/tools.py and utils/patch_tinker.py.
- Rename ambiguous loop var `l` -> `line` (E741) in auto/agent/monitor.py.
- `# noqa: E402` on the intentional late import in twinkle_client/__init__.py.
- setup.cfg per-file-ignores: E501 for auto/agent/monitor.py (embedded LLM
  prompt with verbatim long lines).
- types/__init__.py: scoped `# yapf: disable` around the re-export block so the
  45-name `from .model import (...)` stops oscillating between isort's aligned
  wrap and yapf's hanging wrap (isort still owns the ordering).
@Yunnglin
refactor(client,server): module renames, agent-tools split, and rollo…
f7dc64e 
…ut cleanup

Server:
- Rename utils/validation.py -> utils/auth.py to end the collision with the
  server/validation preflight package (auth/session helpers vs request checks).
- Rename model/utils.py -> model/data_plane_inputs.py to name what it does.

Client:
- Rename http/http_utils.py -> http/client.py and http/utils.py -> http/context.py;
  update all importers.
- Split auto/agent/tools.py into tool_schemas.py (schemas), server_tools.py and
  search_tools.py (ToolExecutor mixins); add test_auto_agent_tools.py.
- Rework rollout/multi_turn.py into an explicit _RolloutState with
  _initialize_state / _process_sequence helpers.

All touched files pass `pre-commit run --all-files` (client now linted).
@Yunnglin Yunnglin changed the title feat(server): server execution guarantees (time bounds + loud failures + contract base) feat(server): async request lifecycle, strict request schema, and execution guarantees Sep 18, 2026
@Yunnglin
update client architecture
e037365
@Yunnglin
fix(model): PEFT 0.18.1 LoRA dtype autocast + static _ensure_lora_dty…
7ba6761 
…pe; repair PPU full test suite

- multi_lora: mirror PeftModel.__init__ adapter dtype autocast on each add_adapter slot (drop unconditional float() normalization)
- transformers: make _ensure_lora_dtype a @staticmethod; update call site
- align tests with PEFT 0.18.1 target-parameter shapes and transport API; loosen slow-startup/backend timeouts
- refresh twinkle client cookbooks/docs for client-as-factory usage
@Yunnglin
feat(server): harden request/error contracts (server-contract-hardeni…
3b7461e 
…ng R6-R13)

- R6: bounded fail-open session liveness via last_liveness_confirmed_at
- R7: full ABC hooks + cluster-global processor lease quota (429/User)
- R8: ConcurrencyError->StateBackendError(503); close() releases handle only,
  flush_all() for teardown; backend contract docstrings
- R9: drop **kwargs pseudo-polymorphism; ModelManager quota -> 429/User
- R10: Twinkle-native ErrorPayload single exit; EndpointUnavailableError moved
  to server/exceptions.py; no-HTTPException static guard
- R11: FutureFailureRecord domain failure; protocol-boundary wire mapping
- R12: client ErrorPayload parse with details/traceback, lowercase category
- R13: remove unrunnable sampler 'torch' option

Verified: unit/contract regression (361 passed) + full 2x2x3 E2E matrix
(transformers+megatron x twinkle+tinker x sft/dpo/grpo, 12/12 passed incl.
save-LoRA/state + resume).
@Yunnglin
refactor(server): module boundaries — server/utils decomposition, Ser…
7062b83 
…verState/BackendGate split (server-module-boundaries Track C)

Structural (behaviour-preserving) module-boundary work:
- server/utils/ dissolved: task_queue/, session_resource/ promoted to server/;
  auth.py -> middleware/auth.py; task_errors.py -> server/; backend_dispatch.py ->
  config/; ray_serve_patch.py -> twinkle/patch/ray_serve.py. device/template utils stay.
- common/ removed: datum.py -> model/tinker_datum.py, router.py -> model/routing.py.
- validation/errors.py merged into deployment.validation_error_handler; validation/
  now holds only backend_compat.
- ServerState split into ResourceCleanupCoordinator + ResourceCountPublisher
  (idempotency guard moved with the impl); actor_name -> cache_key.
- BackendGate extracted from TaskQueueMixin (pure callable wrapper); worker owns
  notify_new_task + the single queue-depth writer.
- tinker_load path resolution moved to the handler; backends take checkpoint_name/
  output_dir and no longer import server.checkpoint.
- gateway services.py -> use_cases.py (dropped one-line forwards); supported_model_names
  public property; per-instance template cache; gateway/routes.py route constants.
- init_twinkle_runtime -> server/runtime.py (with ncpu_proc_per_node passthrough).
- tq_utils.py -> data_format/tq_fields.py (+ TQ field constants; async_rl shim removed).
- lifecycle/protocols.py declares the host contract; run_submit/input_metrics annotated.
- utils bucket export shrunk; telemetry middleware.py split into metrics.py +
  http_middleware.py; dropped stale re-exports (FullModeBusyError, PersistenceConfig,
  TelemetryConfig, _resolve_client_save_dir).

fix(telemetry): per-deployment metric caches moved onto MetricsRegistry instance so
MetricsRegistry.reset() invalidates them (adapters no longer keep NoOp instruments).

Also: added tests (tq_fields, proxy, runtime, protocols, metrics-cache-invalidation),
updated imports/test harnesses, and removed spec-clause reference annotations from code
comments/docstrings.

Verified: regression baseline 291 passed / 16 skipped; full 2x2x3 E2E (transformers &
megatron x twinkle & tinker x sft/dpo/grpo) all passed; flake8/yapf clean.
@Yunnglin
style: satisfy pre-commit lint (isort import order, pyupgrade unquote…
15acc7f 
…s TYPE_CHECKING forward-refs, flake8 line length)
@Yunnglin
style: pyupgrade unquotes forward-ref annotation in state/model_manag…
b00d385 
…er.py (pre-commit --all-files clean)
@Yunnglin
test(server): streamline error payload coverage
b20608f
@Yunnglin
Yunnglin marked this pull request as ready for review September 21, 2026 10:14
Copilot AI lite review requested due to automatic review settings September 21, 2026 10:14
Copilot AI reviewed Sep 21, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5 High severity · 1 Medium severity · 1 Low severity

Open (7)
What changed in this PR

Hardens Twinkle’s server/client request lifecycle with strict shared schemas, bounded execution, persistent futures, and supporting refactors.

Changes:

  • Adds unified protocol models, validation, error payloads, and async submit/retrieve handling.
  • Refactors server state, task queues, telemetry, routing, sampler/model paths, and persistence.
  • Reworks client transport/imports and updates tests, docs, notebooks, and examples.
File Description
tests/​utils/​test_nccl_safe.py Updated as part of this pull request.
tests/​twinkle_client/​test_import_surface.py Updated as part of this pull request.
tests/​twinkle_client/​test_data_plane_async.py Updated as part of this pull request.
tests/​twinkle_client/​test_component_rpc.py Updated as part of this pull request.
tests/​twinkle_client/​test_auto_agent_tools.py Updated as part of this pull request.
tests/​twinkle_client/​test_async_rl_workers.py Updated as part of this pull request.
tests/​twinkle_agentic/​evaluator/​test_client_sampler.py Updated as part of this pull request.
tests/​server/​validation/​__init__.py Updated as part of this pull request.
tests/​server/​utils/​test_task_errors.py Updated as part of this pull request.
tests/​server/​utils/​test_rate_limiter.py Updated as part of this pull request.
tests/​server/​utils/​task_queue/​test_config.py Updated as part of this pull request.
tests/​server/​test_runtime.py Updated as part of this pull request.
tests/​server/​test_deployment_exception_boundary.py Updated as part of this pull request.
tests/​server/​telemetry/​test_metrics_cache_invalidation.py Updated as part of this pull request.
tests/​server/​static/​test_utils_bucket_is_light.py Updated as part of this pull request.
tests/​server/​static/​test_no_twinkle_http_exception.py Updated as part of this pull request.
tests/​server/​static/​test_no_package_root_imports.py Updated as part of this pull request.
tests/​server/​static/​test_client_architecture_imports.py Updated as part of this pull request.
tests/​server/​static/​backend_call_exemptions.py Updated as part of this pull request.
tests/​server/​static/​__init__.py Updated as part of this pull request.
tests/​server/​state/​test_leader_election.py Updated as part of this pull request.
tests/​server/​start_e2e_server.py Updated as part of this pull request.
tests/​server/​sampler/​test_twinkle_async_rows.py Updated as part of this pull request.
tests/​server/​sampler/​test_tinker_handlers.py Updated as part of this pull request.
tests/​server/​sampler/​test_resolve_sampler_weights.py Updated as part of this pull request.
tests/​server/​sampler/​test_mock_sampler.py Updated as part of this pull request.
tests/​server/​model/​test_tinker_handlers.py Updated as part of this pull request.
tests/​server/​model/​test_tinker_compat_output.py Updated as part of this pull request.
tests/​server/​model/​test_replica_lifecycle.py Updated as part of this pull request.
tests/​server/​model/​test_mock_model.py Updated as part of this pull request.
tests/​server/​lifecycle/​test_to_backend_inputs.py Updated as part of this pull request.
tests/​server/​lifecycle/​test_tinker_retrieve_regression.py Updated as part of this pull request.
tests/​server/​lifecycle/​test_protocols.py Updated as part of this pull request.
tests/​server/​lifecycle/​test_envelope_coverage.py Updated as part of this pull request.
tests/​server/​lifecycle/​__init__.py Updated as part of this pull request.
tests/​server/​integration/​test_mock_mode_startup.py Updated as part of this pull request.
tests/​server/​integration/​test_full_param_e2e.py Updated as part of this pull request.
tests/​server/​integration/​test_full_cycle_e2e.py Updated as part of this pull request.
tests/​server/​integration/​test_dpo_e2e.py Updated as part of this pull request.
tests/​server/​integration/​e2e_helpers.py Updated as part of this pull request.
tests/​server/​gateway/​test_proxy.py Updated as part of this pull request.
tests/​server/​gateway/​test_openai_handlers.py Updated as part of this pull request.
tests/​server/​fixtures/​server_config_mock.yaml Updated as part of this pull request.
tests/​server/​data_plane/​test_store.py Updated as part of this pull request.
tests/​server/​data_plane/​test_proxy.py Updated as part of this pull request.
tests/​server/​contract/​update_baseline.py Updated as part of this pull request.
tests/​server/​contract/​test_protocol_migration.py Updated as part of this pull request.
tests/​server/​contract/​test_error_wire.py Updated as part of this pull request.
tests/​server/​conftest.py Updated as part of this pull request.
tests/​server/​config/​server_config_4b_e2e.yaml Updated as part of this pull request.
tests/​server/​config/​server_config_4b_e2e_megatron.yaml Updated as part of this pull request.
tests/​sampler/​test_vllm_startup_lock.py Updated as part of this pull request.
tests/​model/​test_multi_lora_dtype.py Updated as part of this pull request.
tests/​model/​test_micro_batch.py Updated as part of this pull request.
tests/​loss/​test_grpo_gkd.py Updated as part of this pull request.
tests/​data_format/​test_tq_fields.py Updated as part of this pull request.
src/​twinkle/​utils/​import_utils.py Updated as part of this pull request.
src/​twinkle/​server/​validation/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​utils/​task_queue/​config.py Updated as part of this pull request.
src/​twinkle/​server/​utils/​task_errors.py Updated as part of this pull request.
src/​twinkle/​server/​utils/​lifecycle/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​utils/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​telemetry/​worker_init.py Updated as part of this pull request.
src/​twinkle/​server/​telemetry/​tracing.py Updated as part of this pull request.
src/​twinkle/​server/​telemetry/​http_middleware.py Updated as part of this pull request.
src/​twinkle/​server/​telemetry/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​task_queue/​types.py Updated as part of this pull request.
src/​twinkle/​server/​task_queue/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​task_errors.py Updated as part of this pull request.
src/​twinkle/​server/​state/​session_manager.py Updated as part of this pull request.
src/​twinkle/​server/​state/​sampling_manager.py Updated as part of this pull request.
src/​twinkle/​server/​state/​base.py Updated as part of this pull request.
src/​twinkle/​server/​state/​backend/​redis_backend.py Updated as part of this pull request.
src/​twinkle/​server/​state/​backend/​memory_backend.py Updated as part of this pull request.
src/​twinkle/​server/​state/​backend/​factory.py Updated as part of this pull request.
src/​twinkle/​server/​state/​backend/​base.py Updated as part of this pull request.
src/​twinkle/​server/​state/​backend/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​state/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​session_resource/​adapter.py Updated as part of this pull request.
src/​twinkle/​server/​session_resource/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​sampler/​weights.py Updated as part of this pull request.
src/​twinkle/​server/​sampler/​backends/​streaming.py Updated as part of this pull request.
src/​twinkle/​server/​sampler/​backends/​mock_sampler.py Updated as part of this pull request.
src/​twinkle/​server/​sampler/​backends/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​runtime.py Updated as part of this pull request.
src/​twinkle/​server/​processor/​app.py Updated as part of this pull request.
src/​twinkle/​server/​model/​routing.py Updated as part of this pull request.
src/​twinkle/​server/​model/​data_plane_inputs.py Updated as part of this pull request.
src/​twinkle/​server/​model/​backends/​mock_model.py Updated as part of this pull request.
src/​twinkle/​server/​middleware/​auth.py Updated as part of this pull request.
src/​twinkle/​server/​middleware/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​lifecycle/​protocols.py Updated as part of this pull request.
src/​twinkle/​server/​lifecycle/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​launcher/​server_launcher.py Updated as part of this pull request.
src/​twinkle/​server/​launcher/​env_propagation.py Updated as part of this pull request.
src/​twinkle/​server/​gateway/​routes.py Updated as part of this pull request.
src/​twinkle/​server/​gateway/​app.py Updated as part of this pull request.
src/​twinkle/​server/​data_plane/​store.py Updated as part of this pull request.
src/​twinkle/​server/​data_plane/​proxy.py Updated as part of this pull request.
src/​twinkle/​server/​data_plane/​handlers.py Updated as part of this pull request.
src/​twinkle/​server/​config/​server_config.py Updated as part of this pull request.
src/​twinkle/​server/​config/​persistence.py Updated as part of this pull request.
src/​twinkle/​server/​config/​backend_dispatch.py Updated as part of this pull request.
src/​twinkle/​server/​common/​__init__.py Updated as part of this pull request.
src/​twinkle/​server/​checkpoint/​twinkle.py Updated as part of this pull request.
src/​twinkle/​server/​checkpoint/​tinker.py Updated as part of this pull request.
src/​twinkle/​server/​checkpoint/​checkpoint_manager.py Updated as part of this pull request.
src/​twinkle/​server/​checkpoint/​__init__.py Updated as part of this pull request.
src/​twinkle/​sampler/​vllm_sampler/​vllm_sampler.py Updated as part of this pull request.
src/​twinkle/​sampler/​base.py Updated as part of this pull request.
src/​twinkle/​protocol/​types/​training.py Updated as part of this pull request.
src/​twinkle/​protocol/​types/​session.py Updated as part of this pull request.
src/​twinkle/​protocol/​types/​server.py Updated as part of this pull request.
src/​twinkle/​protocol/​types/​processor.py Updated as part of this pull request.
src/​twinkle/​protocol/​types/​checkpoint.py Updated as part of this pull request.
src/​twinkle/​protocol/​json_utils.py Updated as part of this pull request.
src/​twinkle/​protocol/​headers.py Updated as part of this pull request.
src/​twinkle/​protocol/​__init__.py Updated as part of this pull request.
src/​twinkle/​patch/​ray_serve.py Updated as part of this pull request.
src/​twinkle/​model/​transformers/​transformers.py Updated as part of this pull request.
src/​twinkle/​model/​optimizer_group.py Updated as part of this pull request.
src/​twinkle/​model/​multi_lora.py Updated as part of this pull request.
src/​twinkle/​model/​megatron/​multi_lora_megatron.py Updated as part of this pull request.
src/​twinkle/​model/​megatron/​__init__.py Updated as part of this pull request.
src/​twinkle/​model/​__init__.py Updated as part of this pull request.
src/​twinkle/​loss/​grpo.py Updated as part of this pull request.
src/​twinkle/​infra/​_ray/​ray_helper.py Updated as part of this pull request.
src/​twinkle/​dataset/​base.py Updated as part of this pull request.
src/​twinkle/​data_format/​tq_fields.py Updated as part of this pull request.
src/​twinkle/​data_format/​encoding.py Updated as part of this pull request.
src/​twinkle/​data_format/​__init__.py Updated as part of this pull request.
src/​twinkle/​_lazy_module.py Updated as part of this pull request.
src/​twinkle/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​utils/​patch_tinker.py Updated as part of this pull request.
src/​twinkle_client/​utils/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​types/​server.py Updated as part of this pull request.
src/​twinkle_client/​types/​processor.py Updated as part of this pull request.
src/​twinkle_client/​types/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​skills/​modelscope_provider.py Updated as part of this pull request.
src/​twinkle_client/​skills/​manager.py Updated as part of this pull request.
src/​twinkle_client/​skills/​bundled/​twinkle-training.md Updated as part of this pull request.
src/​twinkle_client/​skills/​base.py Updated as part of this pull request.
src/​twinkle_client/​sampler/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​py.typed Updated as part of this pull request.
src/​twinkle_client/​processor/​base.py Updated as part of this pull request.
src/​twinkle_client/​processor/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​model/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​http/​utils.py Updated as part of this pull request.
src/​twinkle_client/​http/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​dataset/​packing_dataset.py Updated as part of this pull request.
src/​twinkle_client/​dataset/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​dataloader/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​common/​remote_component.py Updated as part of this pull request.
src/​twinkle_client/​common/​component_rpc.py Updated as part of this pull request.
src/​twinkle_client/​common/​__init__.py Updated as part of this pull request.
src/​twinkle_client/​auto/​runtime.py Updated as part of this pull request.
src/​twinkle_client/​auto/​connection.py Updated as part of this pull request.
src/​twinkle_client/​auto/​app.py Updated as part of this pull request.
src/​twinkle_client/​auto/​agent/​search_tools.py Updated as part of this pull request.
src/​twinkle_client/​auto/​agent/​monitor.py Updated as part of this pull request.
src/​twinkle_client/​auto/​agent/​core.py Updated as part of this pull request.
src/​twinkle_client/​async_rl/​workers.py Updated as part of this pull request.
src/​twinkle_client/​__init__.py Updated as part of this pull request.
src/​twinkle_agentic/​async_rl/​tq_utils.py Updated as part of this pull request.
src/​twinkle_agentic/​async_rl/​pipeline.py Updated as part of this pull request.
src/​twinkle_agentic/​async_rl/​data_plane.py Updated as part of this pull request.
setup.cfg Updated as part of this pull request.
README.md Updated as part of this pull request.
README_ZH.md Updated as part of this pull request.
pyproject.toml Updated as part of this pull request.
notebook/​short_math_grpo.ipynb Updated as part of this pull request.
notebook/​self_cognition.ipynb Updated as part of this pull request.
notebook/​sample.ipynb Updated as part of this pull request.
notebook/​multi_modal.ipynb Updated as part of this pull request.
notebook/​dpo.ipynb Updated as part of this pull request.
docs/​source_zh/​使用指引/​训练服务.md Updated as part of this pull request.
docs/​source_zh/​使用指引/​服务端和客户端/​Tinker兼容客户端.md Updated as part of this pull request.
docs/​source_zh/​使用指引/​快速开始.md Updated as part of this pull request.
docs/​source_zh/​使用指引/​Qwen3.5最佳实践.md Updated as part of this pull request.
docs/​source_zh/​使用指引/​Embedding训练.md Updated as part of this pull request.
docs/​source_en/​Usage Guide/​Train-as-a-Service.md Updated as part of this pull request.
docs/​source_en/​Usage Guide/​Server and Client/​Tinker-Compatible-Client.md Updated as part of this pull request.
docs/​source_en/​Usage Guide/​Quick-Start.md Updated as part of this pull request.
docs/​source_en/​Usage Guide/​Introduction-with-Qwen3.5.md Updated as part of this pull request.
docs/​source_en/​Usage Guide/​Embedding-Training.md Updated as part of this pull request.
cookbook/​client/​twinkle/​upload_to_hub.py Updated as part of this pull request.
cookbook/​client/​twinkle/​self_cognition.py Updated as part of this pull request.
cookbook/​client/​twinkle/​sample.py Updated as part of this pull request.
cookbook/​client/​twinkle/​multi_modal.py Updated as part of this pull request.
cookbook/​client/​twinkle/​embedding.py Updated as part of this pull request.
cookbook/​client/​twinkle/​dpo.py Updated as part of this pull request.
cookbook/​client/​tinker/​upload_to_hub.py Updated as part of this pull request.
cookbook/​client/​tinker/​self_cognition.py Updated as part of this pull request.
cookbook/​client/​tinker/​multi_modal.py Updated as part of this pull request.
cookbook/​client/​tinker/​dpo.py Updated as part of this pull request.
cookbook/​client/​server/​transformer/​server_config.yaml Updated as part of this pull request.
cookbook/​client/​server/​megatron/​server_config.yaml Updated as part of this pull request.
cookbook/​client/​server/​megatron/​server_config_4b.yaml Updated as part of this pull request.
cookbook/​client/​async_rl/​server_config.yaml Updated as part of this pull request.
cookbook/​client/​async_rl/​client_orchestrated_grpo.py Updated as part of this pull request.
.pre-commit-config.yaml Updated as part of this pull request.
.gitignore Updated as part of this pull request.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/twinkle/protocol/types/model.py Outdated
Comment thread src/twinkle/protocol/types/model.py
Comment thread src/twinkle_client/auto/agent/core.py
Comment thread src/twinkle_client/auto/agent/monitor.py
Comment thread src/twinkle_client/http/client.py
Comment thread src/twinkle/server/config/persistence.py
Comment thread tests/data_format/test_tq_fields.py Outdated
@Yunnglin Yunnglin changed the title feat(server): async request lifecycle, strict request schema, and execution guarantees feat(server): execution guarantees, async lifecycle, and strict wire contracts Sep 21, 2026
tastelikefeet
tastelikefeet reviewed Sep 21, 2026
View reviewed changes
Comment thread src/twinkle/data_format/encoding.py
ENCODED_INPUT_KEYS: tuple[str, ...] = ('input_ids', 'input_embedding')


def is_encoded(entry: Any) -> bool:

@tastelikefeet tastelikefeet Sep 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个看起来不是 data_format，迁移 utils？

@Yunnglin Yunnglin Sep 21, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这是判断输入是不是“模型已经编码好的数据”，属于数据格式逻辑，不是通用工具；放 data_format 可能更合适

tastelikefeet
tastelikefeet reviewed Sep 21, 2026
View reviewed changes
Comment thread src/twinkle/infra/_ray/ray_helper.py
"""
import ray
return ray.get(RayHelper.execute_all_async(method_name, workers_and_args))
return ray.get(RayHelper.execute_all_async(method_name, workers_and_args), timeout=timeout)

@tastelikefeet tastelikefeet Sep 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果增加 timeout，内部 hang 住会不会导致后续请求卡死

@Yunnglin Yunnglin Sep 21, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里机制链路是：

  1. 服务端把调用发给多个 Ray actor，得到一组 future。
  2. ray.get(futures, timeout=T) 最多等 T 秒。
  3. 所有 actor 及时返回，就正常汇总结果；任意 actor 卡住导致整批未完成，就抛出超时异常。
  4. 队列捕获异常，把该请求写成失败终态，客户端轮询 future 时拿到超时错误。
  5. 超时后系统会检查模型还活不活；如果上一条任务其实还卡着，就先拒绝新请求，直接告诉用户“后端忙”，避免大家一起排队卡住。
  6. 等卡住的任务自己结束后，系统会自动恢复，新请求可以继续进来。

核心是：不能保证立刻杀死卡住的 actor，但能保证调用方不会无限等待，也不会让后续请求无止境堆积。默认执行 timeout 是 1800 秒（30 分钟）

tastelikefeet
tastelikefeet reviewed Sep 21, 2026
View reviewed changes
Comment thread src/twinkle/model/megatron/megatron.py
entropies = _outputs.get('entropies', None)
unpacked_logits = _outputs.get('logits', None)
except Exception as e:
# Data processing error (e.g. unpack_packed_sequences dimension mismatch).

@tastelikefeet tastelikefeet Sep 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里去掉 try-except 的原因呢

@Yunnglin Yunnglin Sep 21, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

以前出错会偷偷吞掉，再返回零 loss 假装训练成功；现在改成直接报错，让调用方知道训练确实失败了

tastelikefeet
tastelikefeet reviewed Sep 21, 2026
View reviewed changes
Comment thread src/twinkle/model/optimizer_group.py
_last_grad_norm: float = 0.0

def __setattr__(self, name, value):
if name == 'loss_instance' and value is not None:

@tastelikefeet tastelikefeet Sep 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个去掉的原因呢

@Yunnglin Yunnglin Sep 21, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同上，同样是删掉“出错就用零 loss 顶替”的逻辑

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tastelikefeet tastelikefeet tastelikefeet left review comments
Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Yunnglin @tastelikefeet