Repository navigation
PUT /fetch/manifest/files returns 500 once 13 executions of a manifest have failed #8399
Description
This issue was written by Claude Code.
On prod, on 2026-10-13 at 23:00:57 PDT (2026-10-14 06:00:57 UTC), PUT /fetch/manifest/files?catalog=dcp60&format=verbatim.jsonl&filters={"projectId":{"is":["f83165c5-e2ea-4d15-a5cf-33f3550bffde"]}} from 143.248.75.58 (R06-Public-Metadata-Research/1.0) returned a 500:
{"Code": "ChaliceViewError", "Message": "Too many executions of this manifest generation"}The Lambda request ID was 0acb3b19-b8b6-433c-adaf-c26614d7bcb4. Executions 9a7550e4-5527-5ce8-af0b-c50e54670946_0 to _12 all have the status FAILED, _5 to _12 with the ConnectionTimeout reported on #8259. The last DescribeExecution in that request returned the failure whose Lambda request ID is 2aa628ba-f435-4f9e-8e8b-12f38d4e76f9, the one of _12.
The error comes from src/azul/service/manifest_controller.py:673-678:
sfn_execution_expiration = 90 # Fixed by AWS
max_iteration = sfn_execution_expiration // config.manifest_expiration
…
if iteration > max_iteration:
raise ChaliceViewError('Too many executions of this manifest generation')config.manifest_expiration is 7 (src/azul/__init__.py:244), so iterations 0 to 12 are allowed and 13 is refused. The comment above that code says that in practice this many iterations are not expected, because the manifest key is invalidated every time new code is deployed, which typically happens at least once a week.
On 2026-10-14, b8af492c-0ab9-5cb0-8b97-eb047625ba8b_12 failed at 05:17 PDT and a037f918-7471-5fbd-945f-1f7ff15adf40_12 at 04:35 PDT, the same iteration number as above (the failures are tracked in #8294).
Activity
achave11-ucsc commentedon Oct 14, 2026
This comment was written by Claude Code.
Recurred 5 times in `prod` on 2026-10-14 between 00:04:04 and 04:23:21 PDT (07:04:04 to 11:23:21 UTC), without tripping an alarm, as the `azul-service_5xx-prod` alarm has been in ALARM since 2026-10-13 14:54:14 PDT.
PUT /fetch/manifest/filesfor the same manifest (dcp60,verbatim.jsonl, projectf83165c5-e2ea-4d15-a5cf-33f3550bffde,9a7550e4-5527-5ce8-af0b-c50e54670946) from 143.248.75.58 with the user agentR06-Public-Metadata-Research/1.0returned{"Code": "ChaliceViewError", "Message": "Too many executions of this manifest generation"}at 00:04:04, 01:07:25, 02:09:20, 03:17:11 and 04:23:21 PDT.a037f918-7471-5fbd-945f-1f7ff15adf40_12failed at 04:35:35 PDT, so the manifest for project8c3c290d-dfff-4553-8868-54ce45f4ba7fhas also used all 13 iterations.achave11-ucsc commentedon Oct 15, 2026
This comment was written by Claude Code.
Recurred 65 times in `prod` between 2026-10-14 05:23 and 2026-10-15 11:21 PDT (12:23 on 2026-10-14 to 18:21 on 2026-10-15 UTC). The last three tripped the `azul-service_5xx-prod` alarm at 11:23:14 PDT (18:23:14 UTC).
PUT /fetch/manifest/filesfrom 143.248.75.58 with the user agentR06-Public-Metadata-Research/1.0returned{"Code": "ChaliceViewError", "Message": "Too many executions of this manifest generation"}65 times after the last recurrence listed above. 62 of them were between 05:23:24 PDT on 2026-10-14 and 02:34:46 PDT on 2026-10-15. They didn't trip an alarm, sinceazul-service_5xx-prodwas in ALARM from 2026-10-13 14:54:14 PDT until 2026-10-15 03:36:14 PDT. The alarm tripped again at 11:23:14 PDT because of the last three, at 11:21:17, 11:21:17 and 11:21:23 PDT (18:21:17, 18:21:17 and 18:21:23 UTC):_128cef81db-72af-498d-8bce-30c8432ea7549a7550e4-5527-5ce8-af0b-c50e54670946f83165c5-e2ea-4d15-a5cf-33f3550bffdeConnectionTimeoutc3c8f75b-d609-47dd-aa52-9350690ed28cb8af492c-0ab9-5cb0-8b97-eb047625ba8be0009214-c0a0-4a7b-96e2-d6a83e966ce0Sandbox.Timedout(task timed out after 900 s)ad578725-2c43-4528-8a4a-a84689b3f382a037f918-7471-5fbd-945f-1f7ff15adf408c3c290d-dfff-4553-8868-54ce45f4ba7fRuntime.OutOfMemoryIn each request, the execution for iteration
_12already existed with the status FAILED, and iteration 13 was refused.achave11-ucsc commentedon Oct 15, 2026
@hannes-ucsc: "There needs to be a limit on the number of retried generations for the same manifest. The reason this occurs so frequently right now is that the JSONL manifest has become more popular. Once we fix the underling cause of the generation failures (#8393), we should observe fewer recurrences of this. We may also want to consider a different response code as long as we get some type of alarm notification when ever a manifest generation fails."
achave11-ucsc commentedon Oct 15, 2026
This comment was written by Claude Code.
Recurred 3 times in `prod` on 2026-10-15 at 13:50:38, 13:50:38 and 13:50:41 PDT (20:50:38, 20:50:38 and 20:50:41 UTC), tripping the `azul-service_5xx-prod` alarm at 13:52:14 PDT (20:52:14 UTC).
PUT /fetch/manifest/filesfrom 143.248.75.58 with the user agentR06-Public-Metadata-Research/1.0returned{"Code": "ChaliceViewError", "Message": "Too many executions of this manifest generation"}for the same three generations as in the previous comment:af6e4046-5f37-4558-9c97-b86d7caa17d09a7550e4-5527-5ce8-af0b-c50e54670946f83165c5-e2ea-4d15-a5cf-33f3550bffdea8dd84e2-2731-446d-86d2-318ff1515684b8af492c-0ab9-5cb0-8b97-eb047625ba8be0009214-c0a0-4a7b-96e2-d6a83e966ce08e39d750-4395-4919-aeb2-7b735ea5b249a037f918-7471-5fbd-945f-1f7ff15adf408c3c290d-dfff-4553-8868-54ce45f4ba7fIn each request, the execution for iteration
_12already existed with the status FAILED, and iteration 13 was refused. The alarm had been OK since 12:23:14 PDT.achave11-ucsc commentedon Oct 16, 2026
This comment was written by Claude Code.
Recurred 54 times in `prod` between 2026-10-15 14:56 and 2026-10-16 08:54 PDT (2026-10-15 21:56 to 2026-10-16 15:54 UTC), tripping the `azul-service_5xx-prod` alarm 16 times, from 14:58:14 PDT on 2026-10-15 to 08:55:14 PDT on 2026-10-16.
PUT /fetch/manifest/filesfrom 143.248.75.58 with the user agentR06-Public-Metadata-Research/1.0returned{"Code": "ChaliceViewError", "Message": "Too many executions of this manifest generation"}in 18 rounds, about an hour apart. Each round was three requests within about two seconds, one for each of the generations9a7550e4-5527-5ce8-af0b-c50e54670946(projectf83165c5-e2ea-4d15-a5cf-33f3550bffde),b8af492c-0ab9-5cb0-8b97-eb047625ba8b(e0009214-c0a0-4a7b-96e2-d6a83e966ce0) anda037f918-7471-5fbd-945f-1f7ff15adf40(8c3c290d-dfff-4553-8868-54ce45f4ba7f) from the previous comments.azul-service_5xx-prodALARM (UTC)achave11-ucsc commentedon Oct 19, 2026
This comment was written by Claude Code.
Recurred 25 times in `prod` between 2026-10-16 10:58 and 2026-10-17 10:20 PDT (17:58 on 2026-10-16 to 17:20 on 2026-10-17 UTC), tripping the `azul-service_5xx-prod` alarm at 11:00:14, 12:03:14, 13:07:14, 14:10:14, 15:12:14 and 16:17:14 PDT on 2026-10-16.
PUT /fetch/manifest/filesfrom 143.248.75.58 with the user agentR06-Public-Metadata-Research/1.0returned{"Code": "ChaliceViewError", "Message": "Too many executions of this manifest generation"}in six more hourly rounds of three requests, one for each of the three generations in the previous comments (UTC): 17:58:41, 19:01:32, 20:05:09, 21:08:09, 22:10:04 and 23:15:09.The service Lambda changed from version 44 to 45 at 23:44 UTC. At 00:15 UTC on 2026-10-17 the client's
PUTs started three new generations:74cc5a63-4591-50de-96ee-358e101ae77d(projectf83165c5-e2ea-4d15-a5cf-33f3550bffde),2adabafa-bdb4-58e5-a34c-d1dff8436c9d(e0009214-c0a0-4a7b-96e2-d6a83e966ce0) andee0eec09-3ce2-5478-8c46-b4ddc04ffd61(8c3c290d-dfff-4553-8868-54ce45f4ba7f). Executions_0to_12of each had failed by 15:39 UTC, and the same error then returned for them seven times (UTC). These requests came while the alarm was already in ALARM, from 06:50:14 to 18:22:14 UTC.eaf7c50c…74cc5a63-…00f4c6e5…74cc5a63-…392358a8…2adabafa-…2882c1bb…74cc5a63-…749094e7…ee0eec09-…a0b4eebe…2adabafa-…902c7d45…74cc5a63-…The client's last 5xx was at 17:20:22 UTC.
hannes-ucsc commentedon Oct 22, 2026
(Posted by Claude Code)
Blocker #8393 landed on
developvia PR #8418