Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,11 @@ jobs:

- run: make bootstrap
- run: make check

credential-helper:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- run: python3 -m unittest discover -s scripts -p 'test_*.py'
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@
.DS_Store

.env
__pycache__/
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ We strongly encourage you to write tests for your proposed changes. You can run
make test
```

For the bounded live CLI suite, see [storage acceptance setup](doc/storage-acceptance.md). Live tests require explicit configuration and use renewable profile credentials; ordinary test runs use fake storage.

### Running static code analysis tools

Here are the list of tools that are used to check the sanity of the code at compile time:
Expand Down
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ test_with_race:
test_without_race:
@S5CMD_BUILD_BINARY_WITHOUT_RACE_FLAG=1 go test -mod=readonly -count=1 ./...

.PHONY: acceptance
acceptance:
@S5CMD_TEST_MODE=live go test -mod=readonly ./e2e -run '^TestAcceptanceStorage$$' -count=1 -v -timeout=20m

##@ Bootstrap
# See following issues for why errors are ignored with `-e` flag:
# * https://github.com/golang/go/issues/61857
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ storage services and local filesystems.
- [S3 Transfer Acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html) support
- Google Cloud Storage (and any other S3 API compatible service) support
- Structured logging for querying command outputs
- Virtual-host addressing for custom S3 endpoints with `--use-virtual-host-style`
- Shell auto-completion
- S3 ListObjects API backward compatibility

Expand Down
5 changes: 5 additions & 0 deletions command/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,10 @@ var app = &cli.App{
Name: "use-list-objects-v1",
Usage: "use ListObjectsV1 API for services that don't support ListObjectsV2",
},
&cli.BoolFlag{
Name: "use-virtual-host-style",
Usage: "use bucket subdomains for a custom S3 endpoint",
},
&cli.StringFlag{
Name: "request-payer",
Usage: "who pays for request (access requester pays buckets)",
Expand Down Expand Up @@ -186,6 +190,7 @@ func NewStorageOpts(c *cli.Context) storage.Options {
NoVerifySSL: c.Bool("no-verify-ssl"),
RequestPayer: c.String("request-payer"),
UseListObjectsV1: c.Bool("use-list-objects-v1"),
UseVirtualHostStyle: c.Bool("use-virtual-host-style"),
Profile: c.String("profile"),
CredentialFile: c.String("credentials-file"),
LogLevel: log.LevelFromString(c.String("log")),
Expand Down
56 changes: 56 additions & 0 deletions doc/storage-acceptance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Storage acceptance tests

`make acceptance` builds the current checkout and runs a bounded CLI suite against an explicitly configured storage service. It requires live configuration and never falls back to fake storage. Ordinary `make test` runs the fake-storage regressions without requesting live credentials.

The suite creates one private bucket named `s5cmd-acceptance-<random hex>`. It checks multipart upload, download integrity, metadata, server-side copy, `pipe`, `sync --delete`, ListObjects v1/v2 pagination across 1,001 objects, and wildcard deletion. It uses at most eight fixture workers, two transfer workers, a 15-minute suite deadline, and two-minute command deadlines. Cleanup has a separate three-minute deadline and removes unfinished multipart uploads, object versions, delete markers, and the bucket. Cleanup failures fail the suite.

## Local configuration

Use a dedicated test organization and a restricted AWS profile. Supply actual configuration privately; do not commit it or copy it into PR descriptions.

| Variable | Meaning |
| --- | --- |
| `S5CMD_TEST_MODE` | `live` |
| `S5CMD_TEST_ENDPOINT_URL` | HTTPS storage origin |
| `S5CMD_REGION` | Storage region or availability zone |
| `S5CMD_IS_VIRTUAL_HOST` | `true` |
| `S5CMD_I_KNOW_WHAT_IM_DOING` | `1` to authorize disposable resources |
| `AWS_PROFILE` | Named profile used by fixtures and CLI commands |
| `AWS_CONFIG_FILE` | Private AWS configuration file |
| `AWS_SHARED_CREDENTIALS_FILE` | Private credentials file, empty for process credentials |
| `S5CMD_TEST_RESOURCE_FILE` | Optional new private file recording the exact bucket for recovery |

Unset static AWS credential variables and the legacy `S5CMD_ACCESS_KEY_ID` / `S5CMD_SECRET_ACCESS_KEY` variables. Use a renewable `credential_process` profile; local bootstrap credentials remain local. The fixture client uses the same protected process executor and credential-provider chain as the CLI. Storage TLS verification remains enabled. The suite passes region and endpoint explicitly and uses `--use-virtual-host-style` for custom endpoints.

Run:

```sh
make acceptance
```

The exact test selector is enforced in live mode, so a typo or a subset cannot produce a successful empty live run. Use fresh resource-ledger paths on reruns. The suite reports stage names and selected public error codes; it suppresses command output and raw credential/service diagnostics. Do not enable shell tracing, print environment variables, upload credential caches, or publish resource ledgers.

For a local MinIO rehearsal, retain the existing external-test configuration (`S5CMD_TEST_ENDPOINT_URL`, `S5CMD_REGION`, `S5CMD_IS_VIRTUAL_HOST=false`, both legacy test keys, and explicit consent) and run `go test ./e2e -run '^TestAcceptanceStorage$' -count=1 -timeout=20m` without selecting live mode. This uses synthetic local credentials and path-style addressing.

## GitHub OIDC process helper

`scripts/oidc_credentials.py` is a fallback for a protected GitHub Actions job when a reviewed renewable login action is unavailable. Install a pinned CWIC version supporting `cwic auth accesskey oidc`, then set `CWIC_API_URL`, `CWIC_CACHE_DIR`, and `S5CMD_TEST_ORG` from protected configuration. Use a fresh private cache directory and private AWS config/credentials files. Configure the selected profile with:

```ini
[profile acceptance]
credential_process = python3 /absolute/path/to/scripts/oidc_credentials.py
```

GitHub supplies `ACTIONS_ID_TOKEN_REQUEST_URL` and `ACTIONS_ID_TOKEN_REQUEST_TOKEN` to the job. CWIC calls the helper's token mode on credential exchange and renewal; each token invocation requests a fresh GitHub JWT with audience `https://coreweave.com/iam`. JWTs go only to CWIC; storage credentials go only to the SDK. The helper suppresses CWIC stderr and returns a fixed error on failed login or renewal. It has no API-token fallback.

Run helper regressions with `python3 -m unittest discover -s scripts -p 'test_*.py'`. Normal PR CI runs these synthetic tests without OIDC permission or live configuration.

Storage federation must trust the actual subject issued for this repository's protected environment, including any subject customization. Grant the credential-exchange permission separately from storage permissions on the acceptance bucket prefix. Configure environment reviewers and release-PR verification before granting the execution job `id-token: write`; this harness PR does not enable privileged live workflows or publication. GitHub App authentication for release-bot event delivery is separate from storage federation.

## Cleanup and diagnosis

On ordinary failure, deadline expiry, or a termination signal, the live runner cancels commands and attempts scoped cleanup before exiting. Forced termination or loss of the runner can interrupt cleanup. Keep the private resource ledger available to the trusted cleanup step; recover only its exact bucket with fresh short-lived credentials. Never sweep all buckets or infer ownership from an account inventory. A ledger remains if cleanup fails and is removed on success.

A failed stage, authentication error, or cleanup error blocks the acceptance result. Check private endpoint/region configuration and WIF permissions first; inspect protected service-side diagnostics if needed. Do not expose JWTs, keys, signed requests, or raw HTTP responses in public Actions logs.

Credential renewal, failed renewal, provider precedence, helper sanitization, and retry behavior have synthetic regressions in `storage` and `e2e`; use those to inject failures without intentionally disrupting a live service. Local validation does not verify GitHub-issued subjects or protected-environment enforcement. Release workflow activation and its required gate must validate those separately.
114 changes: 114 additions & 0 deletions e2e/acceptance_cleanup_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
package e2e

import (
"context"
"errors"
"testing"

"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/request"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3iface"
)

func TestSafeStorageErrorCode(t *testing.T) {
for _, code := range []string{"TooManyBuckets", "InvalidRegion", "AccessDenied"} {
if got := safeStorageErrorCode(code); got != code {
t.Fatalf("public storage code %q was suppressed", code)
}
}
for _, code := range []string{"synthetic-private-value", "TooManyBuckets synthetic-private-value", ""} {
if got := safeStorageErrorCode(code); got != "storage request failed" {
t.Fatal("unknown storage code was disclosed")
}
}
}

type cleanupFixture struct {
s3iface.S3API
fail string
bucket string
aborted, deleted, batches int
removed bool
}

func (f *cleanupFixture) check(bucket *string, stage string) error {
if aws.StringValue(bucket) != f.bucket {
return errors.New("cleanup escaped its bucket")
}
if f.fail == stage {
return errors.New("synthetic-private-value")
}
return nil
}
func (f *cleanupFixture) HeadBucketWithContext(_ aws.Context, in *s3.HeadBucketInput, _ ...request.Option) (*s3.HeadBucketOutput, error) {
if f.fail == "absent" {
return nil, awserr.NewRequestFailure(awserr.New("NoSuchBucket", "missing", nil), 404, "synthetic")
}
return &s3.HeadBucketOutput{}, f.check(in.Bucket, "head")
}
func (f *cleanupFixture) ListMultipartUploadsPagesWithContext(_ aws.Context, in *s3.ListMultipartUploadsInput, page func(*s3.ListMultipartUploadsOutput, bool) bool, _ ...request.Option) error {
if err := f.check(in.Bucket, "uploads"); err != nil {
return err
}
page(&s3.ListMultipartUploadsOutput{Uploads: []*s3.MultipartUpload{{Key: aws.String("unfinished"), UploadId: aws.String("upload")}}}, true)
return nil
}
func (f *cleanupFixture) AbortMultipartUploadWithContext(_ aws.Context, in *s3.AbortMultipartUploadInput, _ ...request.Option) (*s3.AbortMultipartUploadOutput, error) {
f.aborted++
return &s3.AbortMultipartUploadOutput{}, f.check(in.Bucket, "abort")
}
func (f *cleanupFixture) ListObjectVersionsPagesWithContext(_ aws.Context, in *s3.ListObjectVersionsInput, page func(*s3.ListObjectVersionsOutput, bool) bool, _ ...request.Option) error {
if err := f.check(in.Bucket, "versions"); err != nil {
return err
}
output := &s3.ListObjectVersionsOutput{}
for n := 0; n < 1001; n++ {
output.Versions = append(output.Versions, &s3.ObjectVersion{Key: aws.String("object"), VersionId: aws.String("version")})
}
page(output, false)
page(&s3.ListObjectVersionsOutput{DeleteMarkers: []*s3.DeleteMarkerEntry{{Key: aws.String("deleted"), VersionId: aws.String("marker")}}}, true)
return nil
}
func (f *cleanupFixture) DeleteObjectsWithContext(_ aws.Context, in *s3.DeleteObjectsInput, _ ...request.Option) (*s3.DeleteObjectsOutput, error) {
f.batches++
f.deleted += len(in.Delete.Objects)
if f.fail == "partial delete" {
return &s3.DeleteObjectsOutput{Errors: []*s3.Error{{Code: aws.String("AccessDenied")}}}, nil
}
return &s3.DeleteObjectsOutput{}, f.check(in.Bucket, "delete")
}
func (f *cleanupFixture) DeleteBucketWithContext(_ aws.Context, in *s3.DeleteBucketInput, _ ...request.Option) (*s3.DeleteBucketOutput, error) {
if err := f.check(in.Bucket, "bucket"); err != nil {
return nil, err
}
f.removed = true
return &s3.DeleteBucketOutput{}, nil
}
func TestAcceptanceCleanup(t *testing.T) {
const bucket = "s5cmd-acceptance-0123456789abcdef01234567"
for _, stage := range []string{"success", "absent", "head", "uploads", "abort", "versions", "delete", "partial delete", "bucket"} {
t.Run(stage, func(t *testing.T) {
fixture := &cleanupFixture{fail: stage, bucket: bucket}
err := cleanAcceptanceBucket(context.Background(), fixture, bucket)
if stage == "success" {
if err != nil || !fixture.removed || fixture.aborted != 1 || fixture.deleted != 1002 || fixture.batches != 2 {
t.Fatal("cleanup omitted resources or exceeded batch size")
}
} else if stage == "absent" {
if err != nil || fixture.removed {
t.Fatal("absent bucket cleanup changed")
}
} else if err == nil {
t.Fatal("cleanup failure must fail the suite")
}
if err != nil && err.Error() == "synthetic-private-value" {
t.Fatal("cleanup exposed service diagnostics")
}
})
}
if err := cleanAcceptanceBucket(context.Background(), &cleanupFixture{}, "another-run"); err == nil {
t.Fatal("invalid cleanup scope accepted")
}
}
Loading
Loading