Settle the producer publish contract + cozy_convert publish robustness (#374, #375)#50
Merged
Merged
Conversation
…avors, no yield shape (#375) Producer endpoints (conversion/dataset/training) write files locally, call cozy_convert.publish_flavors(ctx, flavors) — one Tensorhub /commits commit per ProducedFlavor — and return a result struct. The dead 'yield ProducedFlavor' generator shape is rejected at decoration time so it fails loudly instead of silently streaming and publishing nothing. examples/from-scratch rewritten to the contract with a discovery smoke test; docs updated (endpoint-authoring, decorator + SDK docstrings).
…unk filtering, no silent empty publish (#374) - HubClient: bounded retries with backoff + Retry-After on commit POST, part PUTs, complete, and finalize (429/5xx/network). - run_clone: workdir keyed by (provider, source, destination) under $COZY_CONVERT_WORKDIR; retained on failure so retries resume the snapshot instead of re-downloading multi-GB sources; deleted on success. - files_from_tree/_copy_non_weights skip .cache/huggingface/** so mirrors are byte-faithful to the source repo. - run_clone raises when nothing was published, unconditionally. - Delete the legacy publish_repo_revision second publish path (+ local stub, orphaned helpers/constants); cozy_convert.publish_flavors via /commits is the ONE publish path. Tests: retry + junk-filter coverage in test_hub.py; publish_flavors and run_clone lifecycle (fake /commits server, real HTTP) in test_publish.py. Tracker: #374 + #375 moved to completed.
PaulFidika
force-pushed
the
issue/374-375-publish-contract
branch
from
7cccbb4 to
05bda02
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes tracker issues #375 and #374 (agents/completed.md).
#375 — the ONE publish contract
Producer endpoints (conversion/dataset/training) write files locally, call
cozy_convert.publish_flavors(ctx, flavors)— one Tensorhub/commitscommit perProducedFlavor(path = file or directory) — and return a result struct. The executor stays contract-free.yield ProducedFlavorshape is deleted AND enforced:@endpointrejects generator handlers for producer kinds at decoration time (TypeError pointing atpublish_flavors).examples/from-scratchrewritten to the contract (returnsFromScratchResultwith revision_ids) + discovery smoke test.#374 — cozy_convert publish/clone robustness
$COZY_CONVERT_WORKDIR— retained on failure (retry resumes the HF snapshot; hf local-dir metadata kept for that), deleted on success; partial flavor trees wiped per retry..cache/huggingface/**filtered from published trees (files_from_tree+_copy_non_weights) — mirrors byte-faithful to the source repo (live-observed junk bug).run_cloneraises when nothing was published, unconditionally.publish_repo_revision(~450 LOC, old/publishroute) deleted with its local stub and orphaned helpers —publish_flavorsvia/commitsis the ONE publish path.Tests
uv run --extra dev pytest -q→ 186 passed, 1 skippeduv run --package cozy-convert --extra dev pytest -q -m "not integration"→ 38 passed, 8 deselected; mypy + ruff clean/commitsserver (real HTTP), transient-failure retries, junk filtering, workdir retain/resume/cleanup🤖 Generated with Claude Code