Thanks for the great intro article for monitoring multiple Kubernetes clusters with Thanos!

Did you try other long-term storage solutions for Prometheus such as Cortex, M3DB or VictoriaMetrics? They may be easier to setup and operate than Thanos for multiple Kubernetes clusters, since they accept the collected data from Prometheus instances across Kubernetes clusters via standard Prometheus remote_write protocol. This means you don't need to set up inter-cluster connections between Thanos querier and Thanos sidecards in order to get global querying view (aka single pane of glass) for recently collected metrics, which weren't stored by Thanos sidecars to S3 yet.

Of course you can use Thanos receiver instead of Thanos sidecars and get the same global querying view. But last time I checked Thanos receiver wasn't production-ready.

See also https://faun.pub/comparing-thanos-to-victoriametrics-cluster-b193bea1683