Consider using AWS SDK for CredentialFetcher

[stanhu]

As discussed in #721, the Fog::AWS::CredentialFetcher continues to lag support for IAM access since AWS continues to add functionality.

The latest issue we ran into is that while the AWS SDK allows full configuration of the STS endpoint via the AWS_ENDPOINT_URL_STS (https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html, https://docs.aws.amazon.com/sdkref/latest/guide/ss-endpoints-table.html) environment variable, Fog::AWS::CredentialFetcher only supports regional endpoints:

fog-aws/lib/fog/aws/credential_fetcher.rb

Lines 47 to 52 in 95b1503

	sts_endpoint =
	if ENV["AWS_STS_REGIONAL_ENDPOINTS"] == "regional" && region
	"https://sts.#{region}.amazonaws.com"
	else
	"https://sts.amazonaws.com"
	end

Now obviously we can add support for this environment variable, but this seems pretty inefficient as we have always been behind the curve.

I'd like to propose one of two avenues:

1. If :use_iam_profile is enabled, use Aws::InstanceProfileCredentials to fetch the access key ID, secret access key, and session token.
2. Provide a :credential_fetcher option that allows the client to delegate the fetching of credentials to another class that implements the same signature.

@geemus What do you think?

[stanhu]

Actually, it probably makes sense to use Aws::CredentialProviderChain: https://github.com/aws/aws-sdk-ruby/blob/6f4c71d859a0781be0ec8f3b29a8d4576f3d9d44/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb.

But at this point maybe it's easier to have a Fog implementation that only uses the AWS SDK.

[geemus]

I'm open to discussing options certainly. I think my main concern would be around how much we would need to rewrite/change to get there. If it's mostly adding a dependency and then narrowly using some functionality to get credentials that seems very reasonable. If it is instead rewriting a lot of requests/etc to use the SDK that would be a much harder choice in terms of churn/maintenance/etc.

[stanhu]

@geemus Yeah, I explored using Aws::CredentialProviderChain and just including aws-sdk-core, but unfortunately it appears that:

1. This appears to be a private API. Someone has asked to make this public: Make Aws::CredentialProviderChain part of the public API aws/aws-sdk-ruby#2287, but that issue has been closed.
2. Aws::CredentialProviderChain.new.resolve only works at the moment if you don't supply config options (e.g. region, profile).
3. It appears that the config is using the internal Seahorse::Client::Configuration object.

One approach may be to re-implement Aws::CredentialProviderChain using the related aws-sdk-core classes (e.g. AssumeRoleWebIdentityCredentials, ECSCredentials, InstanceProfileCredentials, etc.). That brings into question about the maintainability of that code as well, but it might be more manageable than what is in fog-aws right now.

Another approach may be to use the service-specific classes, such as aws-sdk-s3:

require 'aws-sdk-s3'
opts = { region: 'us-east-1' }
client = Aws::S3::Client.new(opts)
irb(main):012:0> client.config.credentials
=>
#<Aws::InstanceProfileCredentials:0x000078093caee8f0
 @async_refresh=false,
 @backoff=#<Proc:0x000078093caee710 /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/aws-sdk-core-3.201.4/lib/aws-sdk-core/instance_profile_credentials.rb:145 (lambda)>,
 @before_refresh=nil,
 @credentials=#<Aws::Credentials access_key_id="REDACTED">,
 @disable_imds_v1=false,
 @endpoint="http://169.254.169.254",
 @expiration=2024-09-09 02:11:49 UTC,
 @http_debug_output=nil,
 @http_open_timeout=1,
 @http_read_timeout=1,
 @imds_v1_fallback=false,
 @mutex=#<Thread::Mutex:0x000078093caee6e8>,
 @no_refresh_until=nil,
 @port=80,
 @retries=0,
 @token=#<Aws::InstanceProfileCredentials::Token:0x000078093caf1870 @created_time=2024-09-08 20:06:52.474230093 +0000, @ttl=21600, @value="REDACTED">,
 @token_ttl=21600>

However, you would have to include each specific AWS SDK library rather than just aws-sdk-core.

[geemus]

@stanhu Yeah. Thanks for that research, unfortunately it doesn't sound like it will be particularly easy to do, though as you have pointed out, what we are doing now hasn't been the easiest either in it's own ways.