Timescale recently published Promscale — an open source long-term remote storage for Prometheus built on top of TimescaleDB. According to the announcement, it should be fast and resource efficient. Let’s compare performance and resource usage on production workload for Promscale and VictoriaMetrics. Benchmark setup The following scheme has been constructed for the…

Recently single-node VictoriaMetrics gained support for scraping Prometheus targets. This made possible to run apples-to-apples benchmark, which compares resource usage for Prometheus and VictoriaMetrics during scraping big number of real node_exporter targets. Benchmark setup The benchmark was run in Google Compute Engine on four machines (instances): An instance with node_exporter v1.0.1 for…

Prometheus supports relabeling, which allows performing the following tasks: Adding new label Updating existing label Rewriting existing label Updating metric name Removing unneeded labels Removing unneeded metrics Dropping metrics on certain condition Modifying label names Constructing a label from multiple existing labels Chaining relabeling rules Lets’ looks at how to…

Recently ScyllaDB published an interesting article How Scylla scaled to one billion rows per second. They conducted a benchmark (named Billy) for a typical time series workload, which simulates a million temperature sensors reporting every minute for a year’s worth of data. This translates to 1M*60*24*365=525.6 billion data points. The…

Many technical terms could be used when referring to Prometheus storage — either local storage or remote storage. New users could be unfamiliar with these terms, which could result in misunderstandings. Let’s explain the most commonly used technical terms in this article. Time Series A time series is a series of (timestamp…

It looks like histogram support is great in Prometheus ecosystem: Prometheus client libraries provide Histogram metrics. Prometheus provides histogram_quantile function for calculating quantiles over histogram buckets. Grafana supports visualizing Prometheus histogram buckets via Heatmap panel. But why Prometheus users continue complaining about issues in histograms? Let’s look at the most…

Recently the Evaluating Performance and Correctness article has been published by Prometheus author. The article points to a few data model discrepancies between VictoriaMetrics and Prometheus. It also contains benchmark results showing poor compression ratio and poor performance for VictoriaMetrics comparing to Prometheus. Unfortunately the original article doesn’t support comments…

Suppose you have a time series database containing terabytes of data. How do you mange backups for this data? Do you think it is too big to backup and blindly rely on database replication for data safety? Then you are in trouble. Why replication doesn’t save from disaster? Replication is the process of creating multiple copies…

Thanos is known as long-term storage for Prometheus, while cluster version of VictoriaMetrics had been open sourced recently. Both solutions provide the following features: Long-term storage with arbitrary retention. Global query view over data collected from multiple Prometheus instances. Horizontal scalability. Let’s compare different aspects of Thanos and VictoriaMetrics starting…