Store Gateway OOM

[akshat-kumar-singhal]

Cortex v1.13
Store-gateway memory usage suddenly spiked last week from ~2GB to over 50GB, and now getting OOM issue on the store gateway pod. Not able to identify the root cause for this spike.

[vikasbudhwat]

we are also facing the same issue.
My observation is that index-headers gets synced to local disk in store gateway over period of time and no cleanup happens for old index-headers.

Trying to find way/config to cleanup old index files, may be TTL parameter.

Tied with below parameters, but this didn't work.
bucket_store: sync_dir: /data/tsdb-sync ignore_blocks_within: 5m index_header_lazy_loading_enabled: true index_header_lazy_loading_idle_timeout: 1d bucket_index: max_stale_period: 1h

[akshat-kumar-singhal]

Thanks @vikasbudhwat
Using index_header_lazy_loading_enabled: true reduced the memory usage