Skip to content

main merge - #233

Open
matthewpeterkort wants to merge 28 commits into
mainfrom
development
Open

matthewpeterkort wants to merge 28 commits into
mainfrom
development

Conversation

@matthewpeterkort

@matthewpeterkort matthewpeterkort commented May 7, 2026 •

Copy link
Copy Markdown
Contributor

No description provided.

matthewpeterkort and others added 28 commits May 5, 2026 13:57
Co-authored-by: Matthew Peterkort <33436238+matthewpeterkort@users.noreply.github.com>
* update docs for latest

* document new,non-lfs paths

* extract issues

* refactor lfs out of docs

* update docs

---------

Co-authored-by: matthewpeterkort <matthewpeterkort@gmail.com>
Co-authored-by: Matthew Peterkort <33436238+matthewpeterkort@users.noreply.github.com>
* update git-drs cli to be LFS like

* remove sentinel, address issues

* add design doc

* add new command for migrating records, fix go.mod

* Fix ls-files inventory for hydrated tracked LFS files

* fix cli

* fix tests

* integrate git drs init into remote add

* address a bajillion outstanding issues

* fix tests

* fix test

* bump deps

* fix progress bar output styling

* add business logic for handling git-drs rm command

* fix tests

* bump deps

* clean up

* fix tests

* fix go mod

* fix tests
Removed unnecessary introductory text from developer documentation.

Co-authored-by: Matthew Peterkort <33436238+matthewpeterkort@users.noreply.github.com>
* clean up progress upload

* add progress for git drs pull downloads, update docs

* fix tests

* fix coverage test

* fix bot issues

* bump library deps
* fix wiring to support selecting the bucket in instances where more than one bucket is supported

* fix add url command record templating

* fix copy records

* update deps

* test create new branch use case

* fix lint
* get rid of artifact code

* just push

* update deps from data-client to calypr-cli, fix issue where first time users errored out of remote add command from nonexistant gen3 config in home directory

* take a pass cleaning up dead code from legacy architecture

* regorg packages to be more human readable and cleaner

* add returns to swallowed errors

* cleanup internal packages. organize cmd package code

* update docs, fix CI job

* fix ci

* fix tests

* fix tests

* fix test

* fix test

* beef up pull command

* update deps
* patch ping bug

* add local copy records command. remove file_name field from structs

* update releaser

* fixes an issue where git-drs pull would leave working tree in dirty state by refreshing indices after a pull

* fix tests

* fix tests

* fix test

* fix tests

* fix syfon version

* fix the last failing ci test
* Add macOS code signing to GoReleaser configuration

- Configure code signing for darwin binaries using Apple Developer ID
- Use environment variables for Apple certificate credentials
- Follows same pattern as drs_downloader for consistency

* Add Apple certificate secrets to release workflow

* Add macOS Code Signing Operation Guide

Added a comprehensive operation guide for macOS code signing, including setup, release creation, troubleshooting, and best practices.

* Fix macOS code signing configuration: use binary_signs instead of signs

* Add comprehensive binary verification steps to macOS code signing guide

Added detailed verification commands including:
- Basic signature validation
- Certificate and signer information display
- Hardened runtime verification
- Architecture-specific signature checks
- Expected output examples
- Additional troubleshooting for signature verification failures

* Enhance release workflow with binary verification

Add verification step for signed macOS binaries in release workflow.

* Add support for multiple architectures in goreleaser

* Refactor release workflow for GoReleaser

Refactor GitHub Actions workflow for GoReleaser to include build and sign jobs, and update steps for artifact handling.

* fix release signing workflow

Run GoReleaser release on macOS after importing the Apple certificate so
darwin binaries are signed during the release build before packaging.

Also fix the GoReleaser archives config indentation so archive settings are
top-level instead of being parsed as a second binary_signs entry.

---------

Co-authored-by: Matthew Peterkort <33436238+matthewpeterkort@users.noreply.github.com>
Co-authored-by: matthewpeterkort <matthewpeterkort@gmail.com>
Added 'files' configuration to archives section with 'none: "*"' to exclude
all files by default. This prevents README.md and other files from being
included in the tar.gz archives. The binary is still included as the primary
artifact being released.
* fix: remove invalid files field from archives config

* fix: exclude all files from archives except binary

* fix: include only the built binary in archives

* yet another attempt

* no match pattern
Make git-drs push discover committed pointer blobs from the pushed ref tree
instead of relying on changed-path diffs. This prevents Git refs from being
pushed when matching DRS objects were missed, and lets a later push repair
already-published pointer commits with missing Syfon records.

Also add coverage for space-bearing paths, malformed pointer filtering, and
oid deduplication before upload planning
* adds apple notary

* fix staple hook

* Address PR comments
* adds target to generate 100s of files

* removes lfs dependencies

* address codex review comment

---------

Co-authored-by: Matthew Peterkort <33436238+matthewpeterkort@users.noreply.github.com>
Co-authored-by: matthewpeterkort <matthewpeterkort@gmail.com>
…emote … (#248)

* add support for copying recrods from local git cach to calypr remote instances

* update upload ux to be tolerable

* update tests

* fix add url test

* fix test

* fix bug
…ort (#247)

* TDD: add tests for terra
* Support DRS URI pointers in LFS cache paths
* Adds Terra remote model, type
* Updated git drs ping for Terra
* adds terra ping
* Adds DRS uri <--> OID clarification
* clarifies pointer file usage
* adds sequence diagram
* add-ref contacts the configured source authority
* Document multi-remote add-ref resolution
* documents Terra creds
* adds loadGitConfigOverrides
* adds service-info to ping
* adds anvil poc
* Implement AnVIL/Terra reference workflow
* Fix unknown flag error in git drs command
* fix drs uri validation
* fix clean-filter bug
* Keep hydrated DRS URI references clean.
* adds presentation
* adds anvil manifest
* Add example AnVIL manifest workflow slide
* adds docs/index.html
* docs: propose a unified remote CLI and multi-DRS architecture
* ignore working dirs
* clarified --auth and --credential
* Restrict override reads to repo-local config
* Remove Terra auth and mode keys with the remote
* Match the DRS access header schema
* fix: Destination validation can escape through symlinks
* Fix Terra automatic smudge downloads
* Clarify read-only Terra push behavior
* docs DRS compact identifiers and auth discovery
* Make pulled files read-only for read-only remotes
* Document rollback for accidental read-only DRS edits
* Refresh AnVIL POC gaps and roadmap
* Publish end-user docs and presentation with GitHub Pages
* Define DRS preset catalog ownership and safe overrides
* Implement unified remote CLI and built-in DRS presets
* Update user docs for unified remote presets
* listed preset includes auth type
* Complete generic remote runtime boundary
* Clarify read-only Terra push error
* Support inline and non-first DRS access methods
* Resolve unified Gen3 credential sources at runtime
* Resolve scoped Gen3 buckets at runtime
* Preserve source routing in add-ref pointers
* Validate AnVIL checksums before caching downloads
* Hydrate DRS OIDs through the URI downloader
* Skip DRS URI references during push sync
* Reject unresolved providers during remote add
* Validate generic Gen3 authentication modes
* Track add-ref destinations from repository root
* Fix read-only gitattributes marker
* Persist local DRS metadata for manifest references
* Update documentation for unified remote add.
* Fix AnVIL HTTPS access selection
* 1 of 2 update presentation
* 2 of 2 update presentation
* correct formatting
* more presentation fixes
* yet more presentation fixes
* mkdocs options
* mkdocs options take 2

---------

Co-authored-by: Matthew Peterkort <33436238+matthewpeterkort@users.noreply.github.com>
* initial notes
* Add Globus API authentication and transfer support
* Adds Globus transfers for DRS downloads
* Retry skipped objects on push
* Fall back to DRS access IDs
* Honor usable DRS access methods
* Harden Globus transfer handling
* Normalize GA4GH SHA-256 checksums
* Skip unsupported direct access URLs
* Document transport-neutral Globus workflow
* Remove partial Globus downloads
* Improve access method check error msgs
* Resolve S3 access methods by ID
* Improve documentation for globus drs.AccessMethod
* clarify GIT_DRS_GLOBUS_DESTINATION_COLLECTION
* feat: add SDK-backed Globus login
* docs: explain Globus credential setup
* feat: add DRS access method selection policy
* docs: explain access method selection
* docs: add Globus user integration test plan
* feat: implement shared DRS transfer policy
* docs: explain shared DRS and Globus policy
* feat: materialize and batch Globus collections
* docs: explain Globus collection imports
* test: add Globus user integration harness
* fix: tolerate Globus directory timestamps
* test: streamline Globus tutorial integration workflow
* fix: request Globus collection consent scopes
* feat: support checksum-less Globus URL imports
Detect Globus URLs, expand directory and wildcard inputs, hydrate placeholder pointers, persist real checksums, and batch Syfon checksum lookups.
* test: cover Globus add-url tutorial workflow
Retire tutorial-manifest and verify placeholder OIDs, batched Syfon records, unsigned Globus URLs, and hydrated SHA-256 values.
* docs: explain Globus add-url design
Document URL detection, recursive and wildcard imports, placeholder identity, hydration, Syfon batching, and the optional checksum manifest.
* test: verify Globus wildcard imports
Run the tutorial through a quoted file wildcard and assert the exact three selected paths before hydration.
* test: verify mixed Globus and local data
Add three tracked TSV payloads to the tutorial, verify one project pushes and pulls all six objects, document mixed-source workflows, and allow local Syfon filesystem URLs only for local remotes.
* fix: clean failed Globus batch downloads
Remove every destination in an affected Globus group when submission, task waiting, or verification fails so incomplete cache entries cannot be reused.
* fix: reject add-url destination collisions
Create pointer files exclusively so destination filesystem collisions fail without overwriting an earlier imported member.
* test: print tracked local DRS records
Print each local TSV record from the repository DRS store before push and clarify the mixed-source tutorial output.
* fix: persist placeholder checksums locally
Keep pull read-only, use learned SHA-256 values to revalidate placeholder caches, and publish metadata refinements only through an explicit push.
* test: isolate Globus destination override
Clear the ambient destination collection in the shared-source policy test so a developer login environment cannot change its expected route.
* test: update checksum-less add-url E2E
Verify checksum-less add-url payloads hydrate successfully, persist the learned SHA-256 locally, and publish it to Syfon through an explicit push.
* fix: handle Globus pagination and duplicate downloads
Follow offset pagination for recursive directory imports and submit only one Globus transfer item per cache destination.
* fix: preserve placeholder metadata semantics
Keep authoritative remote metadata when cloned placeholders have no local sidecar, and recognize the placeholder checksum marker during single-file Globus verification.
* fix: merge endpoint-only remote overrides
Overlay clone-local endpoints onto shared generic remotes without discarding provider, authentication, scope, or selection policy.
* fix: preserve token directory permissions
Secure token files without chmodding caller-supplied parent directories.
* docs: publish Globus add-url guide
Stage the Globus add-url page and generated presentation in MkDocs input so strict Pages builds resolve every published link.
* fix: reject hostless HTTP access URLs
Require an HTTP host so access planning resolves a valid access ID or falls back to another advertised method.
* fix: preserve raw Globus question wildcards
Escape question marks in the Globus path before URL parsing so documented single-character wildcard patterns remain intact.
* fix: reject removal of shared remotes
Track shared-policy provenance through local overlays and stop the remote remove command before it mutates configuration.
* resolver: continue fallback when GetAccess fails; lookup: preserve full DRS metadata fields from internal records; globus: validate effective credential in status\n\nFixes:\n- Treat per-method access resolution errors as non-fatal so later methods are tried (AnVIL gs:// then HTTPS fallback).\n- Map Description, Version, CreatedTime, UpdatedTime when converting internal records to DRS objects to avoid dropping metadata.\n- Check the effective Globus credential in 'globus status' by validating the environment or stored token (use Check with environment enabled).
* git: remove residual syfon gitlink and ignore local syfon working tree\n\nStop tracking the accidental syfon submodule pointer and add syfon/ to .gitignore to avoid showing local modifications while keeping the working tree intact.
* ls-files: show object-specific DRS IDs
Fixes #257
* pull: enforce access policy and accept empty objects
Resolve review feedback on #251 by rejecting strict access methods the read-only AnVIL resolver cannot enforce, honoring policy precedence, and treating zero-byte cache entries as complete.
* update presentation
* update minio image
* rework push to do bulk existance check rather than bulk data fetch

* upgrade to 1.26.5

* fix bug related to potential missing oids from project based record lookup

* add a tiny client wrapper for etl pod

* make bulk pull operations more efficient

* Expose Git ref pointer inventory

* Adapt pointer inventory to rework API

* update copy records cmd

* merge

* expand copy records to support local instance as well as local files

* feat(syfon): support current generated APIs

Migrate git-drs to Syfon's current generated API and client packages. Preserve credentials, scope lookup, copy, delete, and download behavior across the updated boundaries.

Pin Docker integration to the tested Syfon commit, including the corrected bulk hash response contract.

* test(addurl): restore PR coverage

* test(addurl): match Syfon checksum error

* fix(pull): refresh expired DRS access URLs

* fix: align rebased branch with modern Syfon APIs

* test: align add-url E2E with placeholder lookup

* test: exercise sandboxed Globus transfer flow

* refactor: remove duplicate transfer and command code

Use Syfon client operations for reusable transfer and bucket behavior. Delete dormant reconciliation and test-only paths, consolidate shared helpers, and remove generated coverage artifacts.

* refactor: remove redundant runtime and command state

* fix: harden transfer and repository boundaries

* chore: remove audit work products and cover push reuse

* test: run Gen3 integration against pinned Syfon

Pin the published Syfon audit-fix commit for backend E2E. Port the Calypr kind suite to build git-drs from the current checkout and Syfon from the shared pin.

* fix: initialize Gen3 credentials and use pullable MinIO fixture

* Fix Syfon add-url inspection and MinIO E2E startup

* test: reproduce hidden pull finalization progress

* fix(pull): show verification and checkout after download

* Fix audited git-drs bugs across paths and transfers

* Make filter memory regression reliable under coverage

* Preserve scoped upload IDs and DRS pointer checksums

* Scan Git history across rev-list output formats
* Fix Gen3 profile setup and keep command supported

* Add directory browsing to ls-files

* Match directory paths in ls-files pointer inventory

* Restore all-pointer ls-files view with -a

* Report live pull transfer progress and wait status

* Show pull phases before transfer bytes arrive

* Avoid repeated pull cache hashing before download

* Resume interrupted pull downloads without hashing partial files

* Show pull SHA verification progress

* Show checkout copy progress and clean canceled stages

* Keep pull active through Git index refresh

* Reuse checkout verification during Git index refresh

* Scope pull index refresh to selected files

* Describe index refresh as a file reread

* update deps

* Skip repeat pull checkout for clean hydrated files

* Resolve single-file pulls once before download

* Verify fresh downloads during checkout copy

* Speed up exact-path pull discovery

* Support directory includes and recursive pull

* Scope pull to current directory and clarify queued downloads

* Benchmark bulk access response processing

* Update Syfon client for multipart batching and expiry protection

* Fetch pinned Syfon commit directly in Docker E2E CI

* Distinguish queued verification from active hashing in pull progress
Co-authored-by: Matthew Peterkort <33436238+matthewpeterkort@users.noreply.github.com>
* Fix Gen3 ping probe and upload progress summary

* Avoid apt mirror stall in Syfon E2E workflow
#271)

Co-authored-by: Matthew Peterkort <33436238+matthewpeterkort@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants