Skip to content

Fix: handle Scaleway InvalidRequest for large file CopyObject #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
matthiasL-scality merged 2 commits into from
Aug 6, 2026
Merged

Fix: handle Scaleway InvalidRequest for large file CopyObject #222

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f2d6de9
Fix: handle Scaleway InvalidRequest for large file CopyObject
matthiasL-scality Aug 6, 2026
13976ce
Fix: tighten large-file detection to avoid generic InvalidRequest match
matthiasL-scality Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions lua/copy_build.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,8 @@ local current_object = 0
-- larger than the threshold directly to multipart copy. Used in tests to
-- exercise the multipart code path without needing actual >5 GB files.
-- In production the variable is unset: attempt CopyObject first and only fall
-- back to multipart when the backend returns EntityTooLarge.
-- back to multipart when the backend returns EntityTooLarge or InvalidRequest
-- (Scaleway uses InvalidRequest for files larger than 5 GB).
--
local copy_size_limit = tonumber(os.getenv("COPY_OBJECT_SIZE_LIMIT"))

Expand Down Expand Up @@ -239,7 +240,8 @@ if copy_size_limit then

else

-- Production mode: CopyObject batch, fall back to multipart on EntityTooLarge.
-- Production mode: CopyObject batch, fall back to multipart on EntityTooLarge
-- (AWS S3) or "copy source is larger" InvalidRequest (Scaleway).
for batch_start = 1, total_number_of_objects, batch_size do
local batch_end = math.min(batch_start + batch_size - 1, total_number_of_objects)
local urls = {}
Expand All @@ -258,7 +260,10 @@ else
ngx.say("[" .. current_object .. "/" .. total_number_of_objects .. "] " .. object .. " ... ")
if object_res.status == 200 then
ngx.say('DONE')
elseif object_res.status == 400 and object_res.body:find("EntityTooLarge", 1, true) then
elseif object_res.status == 400 and (
object_res.body:find("EntityTooLarge", 1, true) or
object_res.body:find("copy source is larger", 1, true)
) then
Comment thread
matthiasL-scality marked this conversation as resolved.
ngx.say("large file, switching to multipart copy")
ngx.flush(true)
local ok, err = multipart_copy(object)
Expand Down
Loading