Promscale vs VictoriaMetrics: measuring resource usage in production
Timescale recently published Promscale — an open source long-term remote storage for Prometheus built on top of TimescaleDB. According to the announcement, it should be fast and resource efficient. Let’s compare performance and resource usage on production workload for Promscale and VictoriaMetrics.
Benchmark setup
The following scheme has been constructed for the benchmark:
/--> VictoriaMetrics
2000 x node_exporter <-- vmagent --|
\--> PromscaleNode_exporter v1.0.1 has been installed on a single e2-standard-4 instance in GCP. It exports real-world resource usage metrics such as CPU usage, memory usage, disk IO usage, network usage, etc. These metrics are usually collected in typical production workloads.
Nginx has been put in front of node_exporter. It has been configured with one second response caching. Nginx was needed for handling load from vmagent (aka lightweight scraper Prometheus metrics), which has been set up for scraping the same node_exporter via 2000 distinct hostnames with 30 seconds scrape interval. This translates to 67 (2000/30) scrapes per second for node_exporter on average. Such a load couldn’t be handled by the node_exporter without caching nginx in front of it.
The following Jinja2 template has been used for generating config file for vmagent:
global:
scrape_interval: 30s
scrape_configs:
- job_name: node_exporter
static_configs:
{% for n in range(2000) %}
- targets: ['host-node-{{n}}:9100']
labels:
host_number: cfg_{{n}}
role: node-exporter
env: prod
{% endfor %}All the collected metrics were pushed by vmagent to both VictoriaMetrics and Promscale in parallel via Prometheus remote_write protocol with the ingestion rate around 40000 samples per second. The total number of unique time series (aka cardinality) was 1.2 millions.
vmagent, VictoriaMetrics v1.49.0 and Promscale v0.1.3 have been installed from Docker images with default settings on distinct e2-standard-16 machines with 16vCPUs, 64GB of RAM and 2.5TB persistent SSD disks per each machine. Promscale was running in a pair with TimescaleDB v2.0.0-rc4-pg12 on the same machine. TimescaleDB configs have been tuned with timescaledb-tune tool. The resulting postgresql.conf file has been updated with the following configs:
synchronous_commit = off
wal_compression = onThese configs were suggested by TimescaleDB developers at their Community Slack chat in order to improve ingestion performance. See the corresponding conversation.
A separate 1TB SSD disk has been allocated for TimescaleDB WAL file. Promscale was started with the following command:
docker run --name promscale -d --net=host timescale/promscale:0.1.3 -db-password=secret -log-level=info -db-ssl-mode=disable -async-acks=trueVictoriaMetrics was started with the following command:
docker run --rm -itd --net=host --name victoria -v `pwd`/data:/data victoriametrics/victoria-metrics:v1.49.0 -storageDataPath=/dataA dedicated machine has been set up for scraping node_exporter metrics from machines with VictoriaMetrics and Promscale during the benchmark. These metrics were used for building resource usage graphs shown below.
The benchmark duration was around 24 hours. The total number of ingested samples per each storage system was around 3.2 billion.
Benchmark results
Memory usage
VictoriaMetrics uses 1.3GB of RSS memory, while Promscale climbs up to 37GB during the first 4 hours of the test and then stays around 30GB during the rest of the test. This means that Promscale needs 28x more RSS memory (37GB/1.3GB) than VictoriaMetrics on production workload.
CPU usage
VictoriaMetrics uses 0.25 of a single CPU core, while Promscale starts with 2.5 CPU cores, then gradually climbs to 16 CPU cores in one hour and then uses all the available CPU cores till the end of the test. This means that VictoriaMetrics can handle at least 64 times higher ingestion rate (16/0.25) compared to Promscale on this machine.
Disk IO usage
VictoriaMetrics writes data to disk at the speed of 1MB/s, while Promscale ramps up to 500MB/s during the first 1.5 hours and then writes data to disk at the speed around 450MB/s. This means that VictoriaMetrics needs 450x less disk write IO bandwidth than Promscale on production workload.
VictoriaMetrics doesn’t read data from disk during the benchmark, while Promscale starts reading the data from disk after the first 3 hours of the test. Then it reads the data at the average speed of 10MB/s with peaks up to 100MB/s.
Disk IOPS usage
Promscale generates around 6000 disk write operations per second with peaks up to 15000 disk write operations per second. VictoriaMetrics consistently generates 35 disk write operations per second. This graph proves that VictoriaMetrics is optimized for low-IOPS disks such as HDD. It can achieve much higher performance than Promscale on SSD disks at the same time.
High number of disk write operations induced by Promscale increases wear leveling for SSD, which may significantly shorten SSD lifetime and reduce its durability.
Disk IO distribution for Promscale between WAL disk and data disk
Graphs above show that the WAL processing takes significant share of disk IO resources at Promscale machine. It looks like there is a big room for optimizations for Promscale+TimescaleDB here. At the same time, the majority of disk IOPS (5000 on average) are spent on data storage. This means that Promscale cannot store data on HDD disks in production. VictoriaMetrics doesn’t use WAL, so it is free from high disk usage issues.
Disk space usage
The graph on disk space usage clearly shows that Promscale compacts stored data every 4 hours. This suggests that data compression is enabled and works as expected at TimescaleDB side.
Promscale uses 150GB of disk space for data and around 5GB of disk space for WAL. VictoriaMetrics uses 1.6GB of disk space at the end of the test. Vmagent ingests 3.2 billion samples to both VictoriaMetrics and Promscale during the test. This translates to the following compression rates:
- Promscale — 46 bytes per sample (
150GB / 3.2 billion samples) - VictoriaMetrics — 0.5 bytes per sample (
1.6GB / 3.2 billion samples)
So, VictoriaMetrics provides 92x better compression rate (46/0.5) compared to Promscale on production data.
Conclusions
While Promscale and TimescaleDB show good performance on synthetic benchmarks such as TSBS according the these search results, they show not so good performance on production workloads as shown above:
- Promscale needs 28x more RSS memory than VictoriaMetrics.
- Promscale needs 64x more CPU than VictoriaMetrics.
- Promscale needs 450x more disk IO bandwidth than VictoriaMetrics.
- Promscale needs a disk with 15000 IOPS, while VictoriaMetrics runs smoothly on a disk with 35 IOPS.
- Promscale needs up to 92x more disk space than VictoriaMetrics (not counting WAL size).
Promscale needs a dedicated disk for WAL in order to get better performance according to TimescaleDB recommendations. Unfortunately this doesn’t help too much as shown on the graphs above.
Data compression in Promscale+TimescaleDB isn’t so great as described in the announcement article.
Compare VictoriaMetrics to Promscale on your production workloads and post results in comments. It is easy setting up Prometheus or vmagent to write scraped metrics to multiple remote storage systems in parallel.
Take a look also at Prometheus vs VictoriaMetrics benchmark.
Update: we open-sourced realistic Prometheus benchmark, which can be used for comparing performance for various Prometheus-compatible storage solutions such as Promscale, M3DB, Cortex and Thanos — see this announcement.
