initial proposal of extending gateway to support an external index

[willscott]

Rendered

[mikeal]

i’d like to see a review approval from someone with more experience working on the gateway, i’m just sure who that would be?

[olizilla]

When I worked on IPFS infra last year we tried to manually ensure that all the gateways and cluster nodes were always connected as peers, so they would send out their wantlist to them when a request came in for a CID they didnt already have. The assumption was that a gateway node in another region might have already fetched a popular block, or that we would likely get requests for content that was stored on one of our large storage clusters.

I just found out that go-ipfs config now supports explicit definition of a set of peers https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#peering so that should be much more reliable now.

Could this proposal be achieved by adding these additional indexes as peers?

We could improve all gateways perf by introducing a LRU aware blockstore; right now gateways are dumping popular content during GC every few hours. It would benefit desktop and server deployments and content discovery in general if all nodes automatically held on to regularly accessed blocks.

We could radically improve the ipfs.io gateway perf by creating a gloabally shared blockstore on some thing like cloudflare kv.

[BigLep]

@mikeal and @olizilla are going to talk verbally about next steps here.