feat: Support custom DNS resolver

[kdn36]

Which issue does this PR close?

Mitigates but does not close #726.

Rationale for this change

This PR enables users of object_store to provide a custom DNS resolver. Use case: the custom DNS resolver may have different properties (e.g., caching), and a different footprint (e.g., substantially smaller) than the default options.

What changes are included in this PR?

Enables override of the DNS resolver as used by the reqwest::HttpClient.

Are there any user-facing changes?

Yes. This PR adds ClientOptions::with_dns_resolver(Arc<dyn reqwest::dns::Resolve>).

[tustvold]

I think this is probably fine, but generally we have tried to keep reqwest types out of our public API, especially in light of things like #724, and the fact we now support pluggable HTTP clients. The fact reqwest now has a resolve2 method, also doesn't really bode well for this API's stability longer-term 😅 It appears to have just been the legacy hyper API repackaged.

What do you think about defining our own resolve trait?

[alamb]

Can someone help me out -- is it possible to use a custom DNS resolver with a custom https://docs.rs/object_store/latest/object_store/client/trait.HttpConnector.html and wrapper around another client?

[alamb]

Marking as draft as I think this PR is no longer waiting on feedback and I am trying to make it easier to find PRs in need of review. Please mark it as ready for review when it is ready for another look

[kdn36]

Apologies for the delay.

The custom client HttpConnector trait has the connect(&self, options: &ClientOptions) method, yet the fields of ClientOptions are not public. So to implement the trait and honor the options (which object_store may set or modify), one would have to extract them one by one and reapply. Technically not impossible, but error-prone and difficult to maintain. Unless I am missing something.

Object_store having its own Resolve trait (details to be worked out) should be viable and would enable object_store not to leak implementation details. This is something I can look into. At first glance, this would be the cleanest way forward for our use case.

Fwiw, the code in the PR is in use already with a lightweight shuffling cache and works as expected.

[kdn36]

The latest commit introduces a local DnsResolver trait, which gets mapped to the reqwest::Resolve trait for the built-in reqwest-based path.

The proposed implementation has been validated end-to-end from our client-side library. Kindly review and let me know any needed changes.