Prometheus vs VictoriaMetrics benchmark on node_exporter metrics

Recently single-node VictoriaMetrics gained support for scraping Prometheus targets. This made possible to run apples-to-apples benchmark, which compares resource usage for Prometheus and VictoriaMetrics during scraping big number of real node_exporter targets.

Benchmark setup

The benchmark was run in Google Compute Engine on four machines (instances):

global:
scrape_interval: 10s
scrape_configs:
- job_name: node_exporter
static_configs:
{% for n in range(3400) %}
- targets: ['host-node-{{n}}:9100']
labels:
host_number: cfg_{{n}}
role: node-exporter
env: prod
{% endfor %}

All the host-node-{{n}} hostnames were pointed to the machine with node_exporter target. They were put in /etc/hosts. This emulated scraping from 3400 node_exporter targets.

The node_exporter has been chosen as scrape target because of the following reasons:

Ubuntu 20.04 has been installed on all the machines mentioned above.

Both VictoriaMetrics and Prometheus were started at the same time for scraping the same target node_exporter. The benchmark duration was 24 hours.

Storage stats

Let’s look at storage stats, which is the same for both VictoriaMetrics and Prometheus:

Benchmark results

Disk space usage:

Image for post
Image for post
Disk space usage: VictoriaMetrics vs Prometheus

Prometheus generates 15GB disk space usage spikes at regular intervals, while VictoriaMetrics generates infrequent and much smaller disk space usage spikes. The maximum spike for VictoriaMetrics is 4GB. Let’s look at final disk space usage stats:

Disk IO usage:

Image for post
Image for post
Disk IO: bytes written per second: VictoriaMetrics vs Prometheus
Image for post
Image for post
Disk IO: bytes read per second: VictoriaMetrics vs Prometheus

Both VictoriaMetrics and Prometheus write data to disk at roughly 2MB/s speed when collecting 280K samples per second. Prometheus generates more disk write spikes with much higher values reaching 50MB/s, while the maximum disk write spike for VictoriaMetrics is 15MB/s.

As for disk read side, Prometheus generates spikes for up to 95MB/s at regular intervals, while the maximum disk read spike for VictoriaMetrics is 15MB/s.

CPU usage:

Image for post
Image for post
CPU usage, vCPU cores: VictoriaMetrics vs Prometheus

Both VictoriaMetrics and Prometheus have similar CPU usage patterns:

Memory usage:

Image for post
Image for post
RSS Memory usage: VictoriaMetrics vs Prometheus

VictoriaMetrics consistently uses 4.3GB of RSS memory during benchmark duration, while Prometheus starts from 6.5GB and stabilizes at 14GB of RSS memory with spikes up to 23GB. These memory usage spikes frequently result in OOM crashes and data loss if the machine has no enough memory or there are memory limits for Kubernetes pod with Prometheus. Fortunately, the test machine had 32GB of RAM, so no crashes have been observed :) Read this article if you want to know technical details on why Prometheus can lose data after unclean shutdown such as OOM crash.

According to the graph above Prometheus requires up to 5.3 times (23GB/4.3GB) more RAM than VictoriaMetrics.

Conclusions

Both Prometheus and VictoriaMetrics can scrape millions of metrics from thousands of targets on a machine with a couple of vCPU cores. This is much better result comparing to InfluxDB or TimescaleDB systems according to these benchmarks.

VictoriaMetrics needs up to 5x less RAM and up to 7x less disk space comparing to Prometheus when scraping thousands of node_exporter targets. This translates to significant infrastructure cost savings.

P.S. If you didn’t use VictoriaMetrics yet, then it is time to try it. VictoriaMetrics is free and open source (including cluster version)— https://github.com/VictoriaMetrics/VictoriaMetrics/ . If you need enterprise features and support, then visit https://victoriametrics.com/ .

Written by

Founder and core developer at VictoriaMetrics

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store