[aevnsvi]

Problem

The deploy API (PUT /pulp/maven/<repo>/<path>) turns every uploaded file into its own
repository version. That has three consequences that get worse as a repository is used:

1. Overlapping deploys fail. The task that adds the file is dispatched with
   immediate=True, deferred=False and the repository as an exclusive resource. If another
   deploy holds the repository at that moment, pulpcore cancels the task ("Resources
   temporarily unavailable.") and the API answers 429. Maven, Gradle and sbt do not retry a
   refused upload, so two pipelines publishing into one repository at the same time fail.
   With 2 parallel uploads one gets 429; with 16, most of them do.
2. Every version is expensive. finalize_new_version regenerates maven-metadata.xml
   (two levels for SNAPSHOTs), prefixes.txt and the affected index pages, and its cost
   grows with the repository: up to seven add_content/remove_content calls each rewrite
   the version's whole content_ids, prefixes.txt scans every artifact's group id twice,
   the index-page lookup expands the whole content list, and the generated files are
   written to storage one after another. On a test deployment adding one file took
   0.8 s on a repository of about 600 units and 2.5 s on one of about 8,500.
3. The number of versions explodes. A publish uploads about 15 files per module (jar,
   sources, javadoc, pom, each with .md5 and .sha1), so a module costs 15 versions.
   Since every version stores the full content_ids array, the versions table grows with
   the square of the number of deployed files: a test repository with 1,330 versions held
   24 MB of id arrays for 3,900 units.

Changes

Three commits, each with its own changelog entry:

1. Let concurrent deploys to one repository wait for their turn. The add task may now
   be deferred, and the request waits for it for up to MAVEN_DEPLOY_TASK_TIMEOUT seconds
   (default 30) before answering 429, as the container registry API does for its uploads.
   A new functional test deploys 8 files at once; it fails with 429 without the change.

2. Make finalizing a repository version cheaper. Packages and metadata are collected
   and applied with one add_content and one remove_content; whether prefixes.txt
   changed is decided from the affected prefixes only; affected index pages are found by
   membership (Exists on _content_relationships(), the same pattern as the content
   handler); metadata and index pages are written to storage in parallel by a
   per-process thread pool (reset in forked task processes, which inherit the pool but
   not its threads). The resulting repository content is unchanged; on its own this saves
   about 10% per deploy.

3. Add deployed files to the repository in batches (opt-in). With
   MAVEN_DEPLOY_BATCH = True, a deploy stores the file and its content unit, records it
   in a new MavenPendingDeploy table and answers 201 at once. A commit_pending_deploys
   task adds everything pending in one version once uploads pause for
   MAVEN_DEPLOY_BATCH_LINGER seconds (default 1), at the latest
   MAVEN_DEPLOY_BATCH_MAX_WAIT seconds (default 10) after the oldest pending file. This
   follows the pending blobs of container pushes and PYTHON_GROUP_UPLOADS. What clients
   see stays the same:

   • the distribution serves a pending file until its batch is committed, and recording it
     invalidates the content cache, so a build can resolve what it has just published;
   • within a batch the newest file deployed to a path wins;
   • a maven-metadata.xml deployed by the client within a batch is kept for its artifact
     or SNAPSHOT version, as it is today when it arrives last in a version of its own.

   Uploads arriving together dispatch a single commit task (a Postgres advisory lock per
   repository around the check and the dispatch). Off by default; without the setting the
   deploy API behaves as after commit 1.

Results on a test deployment

	per file	batched
36 sequential PUTs (sbt-like publish)	33-34 s, 36 versions	4 s, 1 version
16 parallel PUTs	12-13 s, 16 versions	0.8 s, 1 version
file readable after its PUT	at once	at once

A scenario covering releases, a SNAPSHOT, redeploys of a release and a SNAPSHOT, a new
prefix, a one-segment group id and a client-uploaded maven-metadata.xml ends in the same
repository content (all units with their fields, metadata, index pages, prefixes.txt,
timestamps aside) with and without batching: 73 versions without it, 2 with it.

Tests

• The existing functional suite passes in the default mode.
• test_concurrent_deploy.py: parallel deploys all succeed (commit 1).
• test_deploy_batch.py, run when the server has MAVEN_DEPLOY_BATCH on: sequential
  deploys share versions, parallel deploys all land, a file is served before its batch is
  committed, and a client's timestamped SNAPSHOT metadata is kept. A test that a redeploy
  within one batch serves the newer file needs Replace superseded content when redeploying the same coordinates #552 (a redeploy with different bytes keeps
  serving the first file today, batched or not); I will add it once Replace superseded content when redeploying the same coordinates #552 is merged.
• test_mvn_deploy.py waits for the batch instead of expecting 12 versions when batching
  is on.

Related finding, not changed here

The generated version-level maven-metadata.xml lists <value>1.1.0-SNAPSHOT</value>
for SNAPSHOTs deployed under timestamped file names, so Maven cannot resolve them from the
generated file. Today this is masked because the client's own metadata arrives last and is
not regenerated; batching keeps that behaviour explicitly.

I can open a separate issue for it, and split this PR into three if that is easier to
review.