Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.env
76 changes: 19 additions & 57 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
name: ci
on:
workflow_dispatch: {}
pull_request:
branches:
- "coreweave"
push:
branches:
- "coreweave"
permissions:
contents: read
jobs:
build:
strategy:
Expand All @@ -20,8 +25,10 @@ jobs:
name: build (${{ matrix.os }}/go-${{ matrix.go-version }})
runs-on: ${{ matrix.os }}-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: ${{ matrix.go-version }}

Expand All @@ -41,6 +48,8 @@ jobs:

name: test (${{ matrix.os }}/go-${{ matrix.go-version }})
runs-on: ${{ matrix.os }}-latest
env:
GOFLAGS: ${{ matrix.os == 'macos' && '-ldflags=-linkmode=external' || '' }}
services:
minio:
image: ${{ (matrix.os == 'ubuntu') && 'bitnamilegacy/minio:2023.7.18' || ''}}
Expand All @@ -55,8 +64,10 @@ jobs:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: ${{ matrix.go-version }}

Expand All @@ -75,61 +86,12 @@ jobs:

runs-on: ${{ matrix.os }}-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: ${{ matrix.go-version }}

- run: make bootstrap
- run: make check

push_dirty_tag:
runs-on: ubuntu-latest
needs: [qa, test, build]
permissions:
contents: write
actions: write
steps:
-
name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
-
name: Get tag
id: get_tag
run: |
git remote add upstream https://github.com/peak/s5cmd.git
git fetch --tags upstream
VERSION=$(git describe --tags --exclude '*-*' --abbrev=0)
echo "VERSION=$VERSION" >> $GITHUB_ENV
echo "SHORT_SHA=`echo ${GITHUB_SHA} | cut -c1-8`" >> $GITHUB_ENV
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "short_sha=`echo ${GITHUB_SHA} | cut -c1-8`" >> $GITHUB_OUTPUT

-
name: Push dirty tag
id: push_dirty_tag
run: |
git config --global user.name "$(git --no-pager log --format=format:'%an' -n 1)"
git config --global user.email "$(git --no-pager log --format=format:'%ae' -n 1)"
git tag -a "${{ steps.get_tag.outputs.version }}-${{ steps.get_tag.outputs.short_sha }}" -m "Release ${{ steps.get_tag.outputs.version }}-${{ steps.get_tag.outputs.short_sha }}"
git push origin "${{ steps.get_tag.outputs.version }}-${{ steps.get_tag.outputs.short_sha }}"

-
name: Trigger GoReleaser
run: |
curl -X POST \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
https://api.github.com/repos/${{ github.repository }}/actions/workflows/goreleaser.yml/dispatches \
-d '{"ref":"${{ steps.get_tag.outputs.version }}-${{ steps.get_tag.outputs.short_sha }}"}'

-
name: Trigger Docker Build
run: |
curl -X POST \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
https://api.github.com/repos/${{ github.repository }}/actions/workflows/docker.yml/dispatches \
-d '{"ref":"${{ steps.get_tag.outputs.version }}-${{ steps.get_tag.outputs.short_sha }}"}'
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@
/s5cmd.exe
/*.iml
.DS_Store

.env
4 changes: 4 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ Here are the list of tools that are used to check the sanity of the code at comp

`make check` command runs all the checks.

Builds use Go modules pinned by `go.mod` and verified by `go.sum`. Credential process execution and explicit profile safeguards live in `storage`; dependencies are unmodified. After updating dependencies, run `go mod verify` and the credential regressions in `storage` and `e2e`.

### Performance

While adding your changes and testing your changes, it would be good to remember that `s5cmd`'s goal is to be the fastest s3 client with a rich set of functionality. Here are some important things to keep in mind:
Expand All @@ -58,6 +60,8 @@ While adding your changes and testing your changes, it would be good to remember

## Submitting a Pull Request

For this fork, target the `coreweave` branch. CI runs build, test, and QA checks for pull requests and merges into that branch. Merging a pull request does not automatically tag or publish a release.

Once you add your changes and all the tests/checks pass, you can submit your pull request to the `peak/s5cmd` repository. Github will trigger automated tests in Github Actions. All tests and checks will be run on different operating systems including `linux`, `macos` and `windows`.

An `s5cmd` maintainer will review your pull request. They may suggest updates for clarity and style, or request additional unit or integration tests.
Expand Down
12 changes: 6 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,18 @@ endif

.PHONY: build
build:
@go build ${GCFLAGS} ${LDFLAGS} -mod=vendor .
@go build ${GCFLAGS} ${LDFLAGS} -mod=readonly .

.PHONY: test
test: $(TEST_TYPE)

.PHONY: test_with_race
test_with_race:
@S5CMD_BUILD_BINARY_WITHOUT_RACE_FLAG=0 go test -mod=vendor -count=1 -race ./...
@S5CMD_BUILD_BINARY_WITHOUT_RACE_FLAG=0 go test -mod=readonly -count=1 -race ./...

.PHONY: test_without_race
test_without_race:
@S5CMD_BUILD_BINARY_WITHOUT_RACE_FLAG=1 go test -mod=vendor -count=1 ./...
@S5CMD_BUILD_BINARY_WITHOUT_RACE_FLAG=1 go test -mod=readonly -count=1 ./...

##@ Bootstrap
# See following issues for why errors are ignored with `-e` flag:
Expand All @@ -59,11 +59,11 @@ semgrep: ## Run semgrep

.PHONY: vet
vet:
@go vet -mod=vendor ./...
@go vet -mod=readonly ./...

.PHONY: check-fmt
check-fmt:
@if [ $$(go fmt -mod=vendor ./...) ]; then\
@if [ $$(go fmt -mod=readonly ./...) ]; then\
echo "Go code is not formatted";\
exit 1;\
fi
Expand All @@ -79,7 +79,7 @@ check-gomod: ## Check go.mod file

.PHONY: gogenerate
gogenerate:
@go generate -mod vendor ./...
@go generate -mod=readonly ./...

.PHONY: clean
clean:
Expand Down
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ against each matching object, in parallel.
`s5cmd` uses official AWS SDK to access S3. SDK requires credentials to sign
requests to AWS. Credentials can be provided in a [variety of ways](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html):

- Command line options `--profile` to use a named profile, `--credentials-file` flag to use the specified credentials file
- Command line options `--profile` to use a named profile from the shared AWS configuration and credentials files, `--credentials-file` to use a specified static credentials file

```sh
# Use your company profile in AWS default credential file
Expand All @@ -155,6 +155,27 @@ requests to AWS. Credentials can be provided in a [variety of ways](https://docs
s5cmd --credentials-file ~/.your-credentials-file --profile my-work-profile ls s3://my-company-bucket/
```

- Renewable credentials from an AWS shared configuration profile

```ini
[profile renewable]
credential_process = credential-helper
```

```sh
AWS_PROFILE=renewable s5cmd ls s3://example-bucket/
# Explicit profile selection also supports credential_process.
s5cmd --profile renewable ls s3://example-bucket/
```

The helper must return AWS credential-process JSON. When it includes `Expiration`, the SDK refreshes credentials within the running process. Environment access keys take precedence over `AWS_PROFILE`; an explicit `--profile` selects that profile ahead of environment access keys. `--credentials-file` selects static credentials and takes precedence over a credential process.

An explicit profile must configure a credential provider. Missing or empty explicit profiles fail without consulting instance or container credentials. Profiles that explicitly select a credential source retain that behavior.

`--endpoint-url` and `--no-verify-ssl` apply to storage requests. Credential exchanges retain their service endpoints and TLS verification.

Credential-process failures stop the operation without falling back to another provider. Helpers run non-interactively with stdin and stderr suppressed, and stop when the command is cancelled or exits. Raw credential output is excluded from provider errors. Trace logs include request summaries and omit signed HTTP headers.

- Environment variables

```sh
Expand Down
2 changes: 2 additions & 0 deletions command/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,8 @@ func AppCommand(name string) *cli.Command {

// Main is the entrypoint function to run given commands.
func Main(ctx context.Context, args []string) error {
ctx, stopCredentialProcesses := storage.WithCredentialProcessContext(ctx)
defer stopCredentialProcesses()
app.Commands = Commands()

return app.RunContext(ctx, args)
Expand Down
Loading
Loading