Prometheus vs VictoriaMetrics benchmark on node_exporter metrics

Recently single-node VictoriaMetrics gained support for scraping Prometheus targets. This made possible to run apples-to-apples benchmark, which compares resource usage for Prometheus and VictoriaMetrics during scraping big number of real node_exporter targets.

Benchmark setup

The benchmark was run in Google Compute Engine on four machines (instances):

  • Two dedicated e2-highmem-4 instances for Prometheus v2.22.2 and VictoriaMetrics v1.47.0 with the following configs: 4vCPU, 32GB RAM, 1TB HDD persistent disk. Both VictoriaMetrics and Prometheus were run with default configs except of the path to the file with scrape configs (i.e. -promscrape.config=prometheus.yml for VictoriaMetrics and — config.file=prometheus.yml for Prometheus). The prometheus.yml file has been generated from the following Jinja2 template:
global:
scrape_interval: 10s
scrape_configs:
- job_name: node_exporter
static_configs:
{% for n in range(3400) %}
- targets: ['host-node-{{n}}:9100']
labels:
host_number: cfg_{{n}}
role: node-exporter
env: prod
{% endfor %}
  • node_exporter exports real-world metrics (CPU usage, RAM usage, disk IO usage, network usage, etc.) under load, so benchmark results could be extrapolated to production Prometheus setups.

Storage stats

Let’s look at storage stats, which is the same for both VictoriaMetrics and Prometheus:

  • Active time series: 2.8 million
  • Samples scraped and stored: 24.5 billion

Benchmark results

Disk space usage:

Disk space usage: VictoriaMetrics vs Prometheus
  • Prometheus: 52.3GB (32.3GB data plus 18GB WAL). This translates to 52.3GB/24.5 billion samples = 2.1 bytes per sample. This means that Prometheus requires up to 7 times (2.1/0.3) more storage space than VictoriaMetrics for storing the same amounts of data.
Disk IO: bytes written per second: VictoriaMetrics vs Prometheus
Disk IO: bytes read per second: VictoriaMetrics vs Prometheus
CPU usage, vCPU cores: VictoriaMetrics vs Prometheus
  • CPU usage spikes for both systems are related to background data compaction. These spikes are mostly harmless for VictoriaMetrics, while they may result in OOM (out of memory) crashes for Prometheus as explained below. See technical details about background compaction (aka merge) in VictoriaMetrics at these docs.
RSS Memory usage: VictoriaMetrics vs Prometheus

Conclusions

Both Prometheus and VictoriaMetrics can scrape millions of metrics from thousands of targets on a machine with a couple of vCPU cores. This is much better result comparing to InfluxDB or TimescaleDB systems according to these benchmarks.

Founder and core developer at VictoriaMetrics