Create a query cluster

[dili-pk]

I am working on a POC with cortex. My setup is a bit complicated..
I have deployed cortex onto 3 different kubernetes clusters. Now I am deploying a 4th cortex deployment just with querier, query-frontend and distributor.. The intention on 4th cortex deployment is a cluster where I can query all these 3 cluster data at once(either using store-gateway of other 3 or connecting it directly via configuration using join_members in memberlist..
However, this has been unsuccessful and I see multiple issues when I am connecting memberlist service of a different k8s cluster to the query cluster..
I would like to understand if this setup is something that is possible or anyone has ever tried..
Thank you in advace!!

here is the config for query cluster..

cortex.yaml: |
    api:
      prometheus_http_prefix: /prometheus
      response_compression_enabled: true
    auth_enabled: false
    blocks_storage:
      backend: s3
      bucket_store:
        bucket_index:
          enabled: true
        sync_dir: /data/tsdb
      s3:
        access_key_id: XXXXX
        bucket_name: xx-xx-xx
        endpoint: XXXXX
        insecure: true
        secret_access_key: XXXXXXX
      tsdb:
        dir: /data/tsdb-sync
    distributor:
      pool:
        health_check_ingesters: true
      shard_by_all_labels: true
    frontend:
      log_queries_longer_than: 10s
    limits:
      enforce_metric_name: true
      max_query_lookback: 0s
      reject_old_samples: true
      reject_old_samples_max_age: 168h
    memberlist:
      bind_port: 7946
      join_members:
      - 'cortex-memberlist'
      - '<memberlist-headless-svc>:7946' ## of a different cluster
    querier:
      active_query_tracker_dir: /data/active-query-tracker
      store_gateway_addresses: |-
        dns+cortex-store-gateway-headless:9095,dns+<store-gateway-headless-svc>:9095 ## of a different cluster
    query_range:
      align_queries_with_step: true
      cache_results: true
      results_cache:
        cache:
          memcached:
            expiration: 1h
          memcached_client:
            timeout: 1s
      split_queries_by_interval: 24h
    runtime_config:
      file: /etc/cortex-runtime-config/runtime_config.yaml
    server:
      grpc_listen_port: 9095
      grpc_server_max_concurrent_streams: 10000
      grpc_server_max_recv_msg_size: 10485760
      grpc_server_max_send_msg_size: 10485760
      http_listen_port: 8080
    storage:
      engine: blocks
    store_gateway:
      sharding_enabled: false

[jakirpatel]

memberlist based on gossip protocol. I dont think pod ips will be resolved from different k8s. We did same PoC for managed alertmanager.

[dili-pk]

Yeah.. Thanks for the reply.. I kind of figured that this is not possible.