-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
PEP 694: Amend scanning in staged releases and add legacy API changes to use staged releases #5070
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
1469ee8
f4d9632
fab9c5e
75a4595
caacbed
3562b92
679169d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| PEP: 694 | ||
| Title: Upload 2.0 API for Python Package Indexes | ||
| Author: Barry Warsaw <barry@python.org>, Donald Stufft <donald@stufft.io>, Ee Durbin <ee@python.org> | ||
| Author: Barry Warsaw <barry@python.org>, Donald Stufft <donald@stufft.io>, Ee Durbin <ee@python.org>, Cary Hawkins <hawkinscary23@gmail.com> | ||
| PEP-Delegate: Dustin Ingram <di@python.org> | ||
| Discussions-To: https://discuss.python.org/t/pep-694-pypi-upload-api-2-0-round-2/101483 | ||
| Status: Draft | ||
|
|
@@ -29,6 +29,9 @@ Along with standardization, the upload API provides additional useful features s | |
| * "staging" a release, which can be used to test uploads before publicly publishing them, | ||
| without the need for `test.pypi.org <https://test.pypi.org/>`__; | ||
|
|
||
| * entering the publishing session workflow from the existing legacy upload API, so that staging is | ||
| available to publishers before their tooling fully adopts this API; | ||
|
|
||
| * artifacts which can be overwritten and replaced, until a session is published; | ||
|
|
||
| * detailed status on the state of artifact uploads; | ||
|
|
@@ -285,6 +288,28 @@ The unguessable :ref:`stage preview URL <staged-preview>` is a separate capabili | |
| governed by this authorization check; it grants read-only preview access to any client that holds the token, | ||
| so that (for example) a CI job can install-test a staged release without project upload credentials. | ||
|
|
||
| As one such stricter policy, an index **MAY** require *additional* authorization, beyond upload permission, for | ||
| the actions that decide a session's terminal outcome, namely :ref:`publishing <publishing-session-completion>` | ||
| it and :ref:`canceling <publishing-session-cancellation>` it, while still allowing session creation and file | ||
| upload with upload permission alone. This separates the authority to assemble a release from the authority to | ||
| publish or discard it: an automated system can hold a credential that creates a session and uploads files to it | ||
| but can neither publish nor cancel it, with that authority held elsewhere, so that compromise of the automated | ||
| system alone can neither publish a release nor discard a pending one. | ||
|
|
||
| Cancellation is protected because it is destructive: it discards the staged files and frees the name-version | ||
| pair. An index that left it at the upload-permission level would let a compromised upload credential delete a | ||
|
cjames23 marked this conversation as resolved.
|
||
| release a maintainer has staged and is waiting to publish (a denial-of-release), or discard a session that | ||
| has been flagged and is awaiting review. For a first release of a new project, cancellation also releases the | ||
| temporary project-name reservation the index holds for the session, so an index that did not protect it would | ||
| let a compromised upload credential discard the session and allow the name to be claimed by someone else. An | ||
| index that instead wants an automated uploader to be able to abort its *own* in-progress uploads **MAY** leave | ||
| cancellation at the upload-permission level. | ||
|
|
||
| How any additional authorization is expressed, and whether it is offered at all, is determined by the index | ||
| operator. An index **MAY** keep this choice operator-wide, or expose it to individual project owners, for | ||
| example by letting a project owner require the separate authorization for their own project, or allow a single | ||
| principal to hold both. | ||
|
|
||
|
|
||
| .. _session-errors: | ||
|
|
||
|
|
@@ -336,9 +361,14 @@ interpretation to aid in diagnosing underlying issue. | |
| Some responses may return more specific HTTP status codes as described in the text below. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note to self. Once PEP 847 is approved, we should update this PEP to specifically reference it. @woodruffw @dstufft |
||
|
|
||
| .. _publishing-session: | ||
| .. _session-creation: | ||
|
|
||
| Session Creation | ||
| ---------------- | ||
|
|
||
| Publishing Session | ||
| ------------------ | ||
| A publishing session can be created in two ways: directly through this API, described below, or from a | ||
| :ref:`legacy upload <legacy-interop>`. However it is created, the session is then managed through the same | ||
| endpoints, described in :ref:`Managing a Publishing Session <managing-session>`. | ||
|
|
||
| .. _publishing-session-create: | ||
|
|
||
|
|
@@ -585,6 +615,73 @@ sub-mapping with the following keys: | |
| these notices are specific to the referenced file. | ||
|
|
||
|
|
||
| .. _legacy-interop: | ||
|
|
||
| Create a Publishing Session from a Legacy Upload | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| Publishers cannot use the features of this API until their upload tooling adopts it, and the legacy API is | ||
| expected to remain available anyway, as this PEP does not propose a deprecation schedule. To make staging available to those publishers sooner, | ||
| an index **MAY** allow a legacy upload to create a publishing session, so that from creation onward the session is the | ||
| same as one created directly. | ||
|
|
||
| An index that supports this **MUST** document it, and **MUST** accept a ``staged`` field with the value | ||
| ``true`` in the legacy ``multipart/form-data`` upload request. When that field is present, the index creates a | ||
| publishing session in the ``open`` state for the uploaded file's project and version, adds the file to it as a | ||
| :ref:`completed <file-upload-session-states>` file upload, and does not publish it. The index **MUST** | ||
| return the :ref:`publishing session creation response body <publishing-session-response>` from that upload, | ||
| including the ``links`` and ``session-token`` keys, so that the publisher can then use the endpoints in this | ||
| PEP to :ref:`preview <staged-preview>`, :ref:`publish <publishing-session-completion>`, or :ref:`cancel | ||
| <publishing-session-cancellation>` the session. Returning the body is required because a client that sets | ||
| ``staged=true`` but receives no session handle would be left unable to publish or cancel the staged release. | ||
| Note that a client that sets ``staged=true`` without understanding the response may record it (for example in | ||
| public CI logs); the ``session-token`` it contains grants only read-only :ref:`preview <staged-preview>` | ||
| access, but indexes and publishers **SHOULD** be aware that it can be exposed this way. An index **MAY** also create a session for an upload based on | ||
| its own policy or the project's configuration, without the field being present; this allows a project to | ||
| require that its releases are staged in a way that an upload client cannot bypass. In this policy driven | ||
| case no client change is required at all; the ``staged`` field lets a publisher opt in per upload with only a | ||
| minimal change to existing tooling. | ||
|
|
||
| Because the legacy API uploads a single file per request, subsequent legacy uploads for the same project and | ||
|
cjames23 marked this conversation as resolved.
|
||
| version **SHOULD** be added to the same open session when that session was itself created through a legacy | ||
| upload, so that the release is still published as a unit. Once such a session exists, a subsequent legacy | ||
| upload for that project and version is added to it whether or not it carries the ``staged`` field: the field | ||
| is not required again, and the index **MUST NOT** reject the upload for omitting it. The field is still | ||
| permitted on these uploads as an explicit statement of intent. | ||
|
|
||
| A session's creation path is fixed when it is created, and the two paths **MUST NOT** be mixed. | ||
| If a non-terminal session already exists for a name-version pair (see :ref:`publishing-session-multiple`), a request to | ||
| contribute to it through the *other* path **MUST** be rejected with a ``409 Conflict``: any legacy upload for | ||
| a pair that already has an open session created through the Upload 2.0 API is rejected (whether or not it | ||
| carries the ``staged`` field), and an Upload 2.0 request that would add to a session created through a legacy | ||
| upload is likewise rejected. | ||
|
|
||
| Fixing the path at creation avoids ambiguity in how the session reaches a terminal state. An Upload 2.0 | ||
| session is driven to completion by a client that holds its ``session-token`` and calls the control plane | ||
| endpoints, whereas a legacy created session is completed by continued legacy uploads and by the index acting | ||
| on the publisher's behalf. Allowing a legacy ``staged=true`` upload to join an existing Upload | ||
| 2.0 session would mean issuing a ``session-token`` for, and granting completion authority over, a session the | ||
| legacy uploader did not create; allowing an Upload 2.0 request to add to a legacy created session would | ||
| subject a legacy publisher to a completion model its tooling does not implement. Rejecting cross path | ||
| contribution keeps the authority to publish or cancel a session with the path that created it, which also | ||
| keeps any :ref:`separate publishing authorization <authentication>` on that session unambiguous. | ||
|
|
||
| A legacy client that is unaware of this PEP cannot issue a :ref:`publish request | ||
| <publishing-session-completion>`. Where an index has created a session on such a client's behalf, and the | ||
| session is subject only to automated processing, the index **MAY** publish the session itself once that | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the scenario where an index uses sessions by policy of the index/project, it seems this leaves clients who do not support the new protocol in bit of a lurch, unable to control when their staged files enter the processing state which may be before they want, or long after. I assume there is some desire for this to eventually be the default on PyPI to provide a mechanism for pre-publication malware scanning, so I think it is a bit unwieldy to not have a story around how we will handle users of legacy clients who may never adopt explicit session support. If I'm honest, I would prefer that we move very quickly to rollout Upload 2.0, and then very quickly to deprecate the legacy endpoint, as active uploaders are the some of the most easily "reached" clients we have. When their publication flows break, they are much more likely to be actively ready to repair them. I'm a bit concerned that having the interop period, with itself potentially being divided between "opt-in" and "by policy" staging, creating more than one distinct migration/adoption period.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tend to agree I would definitely like the rollout of Upload 2.0 and deprecation of the legacy endpoint to be quick in succession. I think otherwise places a lot of burden on maintenance and as you stated it can create more than one adoption period. And speaking from JOB1 experience the longer that a migration is allowed to take the longer that the long tail becomes. |
||
| processing resolves without an adverse result, or once the period allowed for it elapses (see | ||
| :ref:`publishing-session-completion`). An index **MUST NOT** publish a session this way if the publisher has | ||
| configured the project to require a separate publishing authorization (see :ref:`authentication`). | ||
|
|
||
|
|
||
| .. _managing-session: | ||
|
|
||
| Managing a Publishing Session | ||
| ----------------------------- | ||
|
|
||
| The endpoints in this section apply to a publishing session regardless of how it was created (see | ||
| :ref:`session-creation`). | ||
|
|
||
| .. _publishing-session-states: | ||
|
|
||
| Publishing Session States | ||
|
|
@@ -743,6 +840,42 @@ change: deferred processing resolves to either ``published`` on success or ``err | |
| resolves to ``error``, the session remains editable and the reason is reported in the session's ``notices``, | ||
| as described in :ref:`publishing-session-states`. | ||
|
|
||
| Reviewing an upload can produce two kinds of result, surfaced at different points. Correctness problems that | ||
| can be decided about a single file in isolation, such as its size, checksum, metadata, or compatibility tags, | ||
| are surfaced when that :ref:`file upload session <file-upload-session-states>` completes, moving the file to | ||
| the ``error`` state, from which it cannot be repaired in place. Results that concern whether the release may | ||
| become public, such as malware detection, instead gate the publish step and are described below. An index | ||
| **MAY** run malware review per file, as each file is uploaded, as well as (or instead of) at publication; a | ||
| file remains uploadable to a stage for some time before the release is published, so reviewing each file as it | ||
| arrives can shorten the window in which an unreviewed file sits in a stage. | ||
|
|
||
| The ``processing`` state **MAY** be used to run asynchronous review of a session's files before it is | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is the intention that an index may perform similar asynchronous review per file as well or do we only expect such review to occur once a user requests publication of the entire release? The processing state for file uploads (initiated by the caller submitting to the completion endpoint) was kind of where I had this processing occurring in my head. With more thought towards this PR, I think that there's a good argument for per-file and per-release processing, although I think I ultimately like gating on the results being part of the publication step. An index may have an adverse result based on reviews around malware for a given file that is only exposed once publication is attempted, but any processing as it relates to correctness (file size, checksum, metadata, compatibility, etc...) is probably best raised immediately in the file session and treated as non-recoverable as it is currently written. This is mostly me talking it out, but the question I think I'm raising now is: Do we want to have a similar callout for what kinds of results are raised in the file-upload-session completion versus the release publication step?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That is a good call out, I should definitely add something around what kinds of result are raised in the different steps. I think in terms of the malware types of scans those definitely have to be per file just given that iirc there is a 14 day window for file uploads to a given version as of now. Since the primary goal of this type of scanning is to eliminate as many supply chain attacks as possible. |
||
| published, such as malware scanning. If the review completes without an adverse result, the session resolves | ||
| to ``published`` as normal. If the review does not complete within a period chosen by the index, the server | ||
| **MAY** treat it as though it had completed without an adverse result and publish the session, so that a | ||
| backlogged or unavailable review system does not indefinitely prevent publication. If the review produces an | ||
| adverse result, the session resolves to ``error`` with the reason reported in the session's ``notices``; the | ||
| server **MAY** decline to publish such a session on any subsequent retry, in which case it is eventually | ||
|
cjames23 marked this conversation as resolved.
|
||
| :ref:`canceled <publishing-session-cancellation>` and its data discarded. Where a session has resolved to | ||
| ``error`` because of an adverse result, an index **MAY** provide a way for the publisher to request that the | ||
| result be re-examined. Such a re-examination **SHOULD** be performed by a human, to confirm whether the | ||
| adverse result was a false positive; if it was, the index **SHOULD** allow the session to be published. The | ||
| review itself, including which checks run, how long they are allowed to take, and how such requests are | ||
| handled, is determined by the index operator and is out of scope for this specification. | ||
|
|
||
| Because a session remains editable in the ``open`` and ``error`` states, its files can change after a review | ||
| has run. A server **MUST NOT** publish a session on the strength of a review of content the session no longer | ||
| contains: if any file is added, replaced, or deleted after a review, the prior result is invalidated and the | ||
| session **MUST** be reviewed again before it can be published. How an in-progress review reacts to such a | ||
| change is left to the index: it might run to completion, rescan only the changed files, or restart. | ||
|
|
||
| Because editing a session can trigger re-review, an index **SHOULD** guard against abuse of the review system. | ||
|
cjames23 marked this conversation as resolved.
|
||
| For example, a client might repeatedly upload a file it knows will be flagged, delete it, and re-upload it to | ||
| force repeated scans. Mitigations such as rate-limiting reviews, capping the number of review attempts for a | ||
| session, or moving a persistently adverse session to a terminal state are left to the index. An index can | ||
| also avoid this class of abuse entirely by performing the review only at publication time, when the session's | ||
| files are fixed for the duration of the publish and cannot be swapped out to force another scan. | ||
|
|
||
| A publish attempt that fails *synchronously* (i.e. within the publish request itself) is returned to the | ||
| client as an :ref:`error response <session-errors>` and leaves the session in its current editable state; it | ||
| does **not** move the session to ``error``. | ||
|
|
@@ -1847,6 +1980,22 @@ as experience is gained operating Upload 2.0. | |
| Change History | ||
| ============== | ||
|
|
||
| * `01-Oct-2026 <https://discuss.python.org/t/pre-pep-staged-releases-separated-from-pep-694/107804/58>`__ | ||
|
|
||
| * Add :ref:`Legacy Upload API Interoperability <legacy-interop>`, allowing a legacy upload to create a | ||
| publishing session (via a ``staged`` field, or by index or project configuration that an upload client | ||
| cannot bypass) so that staging is usable before upload tooling adopts this API, and allowing an index to | ||
| publish such a session itself once automated processing resolves or its window elapses. Require that an | ||
| index supporting this accept the ``staged`` field and return the session creation response body, specify | ||
| that subsequent legacy uploads to an open stage need not repeat the field, and forbid mixing the two | ||
| session-creation paths. | ||
| * Note that the ``processing`` state **MAY** be used for asynchronous review such as malware scanning, with | ||
| an index-chosen window after which the review is treated as having produced no adverse result. Distinguish | ||
| per-file correctness results from release-level review, allow review per file or at publication, and note | ||
| that reviewing at publication time avoids re-scan abuse. | ||
| * Allow an index to require additional authorization, beyond upload permission, to publish or cancel a | ||
| session, so that duties can be separated, optionally exposing that choice to project owners. | ||
|
|
||
| * `29-Jul-2026 <https://discuss.python.org/t/pep-694-pypi-upload-api-2-0-round-4/108320>`__ | ||
|
|
||
| * Add an **Atomic Publication and Conflicts** section. Specify that publication is atomic with respect to | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.