Context: I have a multi-instance Prometheus stateful set running in my Kubernetes Cluster. I deployed Prometheus with the Thanos sidecar enabled in the HelmRelease managing my Prometheus install. I have also deployed Thanos Querier and Thanos Store Gateway. Thanos querier works as intended, providing a unified web interface that can query across both my prometheus instances and automatically handles deduplication of data.
StoreGateway however only partially works. I have verified that some data is backed up to S3 via the sidecar and StoreGateway is able to successfully query all data that the sidecar backed up from the moment it was deployed.
Problem: My prometheus instance was set up to retain 1yr of data but Thanos StoreGateway + Sidecar is only backing up data from the past couple months (from when the Thanos sidecar was installed).
Should Thanos be backing up data from even before Thanos sidecar was installed?
