How to use relabeling in Prometheus and VictoriaMetrics

Photo by Luke Chesser on Unsplash

Prometheus supports relabeling, which allows performing the following tasks:

• Adding new label
• Updating existing label
• Rewriting existing label
• Updating metric name
• Removing unneeded labels
• Removing unneeded metrics
• Dropping metrics on certain condition
• Modifying label names
• Constructing a label from multiple existing labels
• Chaining relabeling rules

Lets’ looks at how to perform each of these tasks.

Adding new label

New label can be added with the following relabeling rule:

- target_label: "foo"
  replacement: "bar"

This relabeling rule adds {foo="bar"} label to all the incoming metrics. For example, metric{job="aa"} will be converted to metric{job="aa",foo="bar"}.

Updating existing label

Existing label can be updated with the relabeling rule mentioned above:

- target_label: "foo"
  replacement: "bar"

This rule rewrites metric{foo="aaaa"} with metric{foo="bar"}.

Rewriting existing label

The following relabeling rule can be used for removing port part from instance label:

- source_labels: [instance]
  regex: "([^:]+):.+"
  target_label: "instance"
  replacement: "$1"

This rule translates foo{instance="bar:123"} to foo{instance="bar"}.

How does it work: it extracts instance label value (see source_labels list above), applies the given regex to it, then generatesreplacement string ($1 is substituted by the part of instance label that is matched by regex part in the first parenthesis), and then puts the replacement string into target_label.

Updating metric name

Metric name can be updated with the following relabeling rule:

- source_labels: [__name__]
  regex: "(.+)_suffix"
  target_label: "__name__"
  replacement: "prefix_$1"

This rule removes _suffix from metric name and adds prefix_ to it. For example, foo_suffix{bar="aaa"} metric would be substituted with prefix_foo{bar="aaa"} with this relabeling rule.

VictoriaMetrics provides additional action — replace_all, which can be used for replacing all the occurrences of the given pattern with something else. For example, the following relabeling rule substitutes all the . chars with _ chars in metric names:

- action: replace_all
  source_labels: [__name__]
  target_label: "__name__"
  regex: "\\."
  replacement: "_"

Graphite-like metric foo.bar.baz would be substituted with foo_bar_baz after applying this relabeling rule.

Removing unneeded labels

Sometimes it is needed to remove certain labels before storing metrics in Prometheus or VictoriaMetrics. This can be done with action: labeldrop. The name of the label to drop must be specified as a regular expression in regex . For example, the following relabeling rule drops all the label names starting with foo:

- action: labeldrop
  regex: "foo.+"

This relabeling rule transforms the following metric:

metric{job="a",instance="xyz",foobar="baz",foox="aaa"} 123

into:

metric{job="a",instance="xyz"} 123

Sometimes it is needed to drop all the labels except of a few label names matching the given regexp. This can be done with action: labelkeep. The following relabeling rule drops all the labels except of __name__ and labels ending with keepme:

- action: labelkeep
  regex: "__name__|.*keepme"

This means that metric{job="aa",foo="bar",letskeepme="aaa"} would be translated to metric{letskeepme="aaa"} with the given relabeling rule.

Removing the specific label value

The following rule removes foo="bar" label value:

- source_labels: [foo]
  regex: "bar"
  target_label: foo
  replacement: ""

For example, metric{foo="bar",baz="x"} becomes metric{baz="x"} after applying this relabeling rule, while metric{foo="xxx"} remains the same.

Removing unneeded metrics

Metrics can be dropped with action: drop. For example, the following relabeling rule drops metric if it contains instance label starting from foobar:

- action: drop
  source_labels: [instance]
  regex: "foobar.+"

This relabeling rule drops the following metrics:

foo{instance="foobar1"}
foo{instance="foobar2",job="xxx",aaa="bb"}

It doesn’t drop the following metrics:

foo{instance="xxx"}
foo{instance="abc",job="xyz"}

Sometimes it is easier to specify metrics that needs to be preserved instead of metrics that needs to be dropped. In this case action: keep must be used:

- action: keep
  source_labels: [job]
  regex: "foobar"

This relabeling rule preserves metrics with job label equal to foobar , while other metrics will be dropped.

How to drop metrics if they contain certain values for multiple labels? Just enumerate these labels in source_labels and then specify the desired regex. Then source_labels contains multiple labels, they are concatenated with ; char before matching the provided regex. For example, the following relabeling rule would drop metric with {job="foo",instance="bar"} labels:

- action: drop
  source_labels: [job, instance]
  regex: "foo;bar"

Dropping metrics on certain condition

Sometimes it is necessary to drop a metric if it contains two labels with identical values. This can be done with drop_if_equal action, which is supported by VictoriaMetrics and vmagent. For example, the following relabeling rule would drop metric if it contains identical label values for real_port and needed_port:

- action: drop_if_equal
  source_labels: [real_port, needed_port]

The rule would drop the following metric: foo{real_port="123",needed_port="123"} , but would keep the following metric: foo{real_port="123",needed_port="456"}.

VictoriaMetrics also provides keep_if_equal action, which drops metric if its source_labels aren’t equal. This may be useful for filtering out superfluous scrape targets detected by kubernetes_sd_config — see these docs for details.

Modifying label names

Sometimes it is necessary to modify label names. Then action: labelmap can be used for this. For example, the following relabeling rule substitutes foo_ prefix in all the label names with bar_ prefix:

- action: labelmap
  regex: "foo_(.+)"
  replacement: "bar_$1"

Note that the rule leaves the original label untouched in the metric. I.e. the following metric — aa{foo_xx="bb",job="qq"} — is translated to aa{foo_xx="bb",bar_xx="bb",job="qq"} by this relabeling rule.

VictoriaMetrics supports additional action — labelmap_all — which allows substituting specified patterns in label names. For example, the following relabeling rule would substitute all the — chars with _ chars in all the label names:

- action: labelmap_all
  regex: "-"
  replacement: "_"

This rule translates foo{a-b="x",qwe-x-zz="aa"} with foo{a_b="x",qwe_x_zz="aa"}.

Constructing a label from multiple existing labels

Suppose a metric has host and port labels, while you need constructing address label with host:port contents. This can be done with the following relabeling rule:

- source_labels: [host, port]
  separator: ":"
  target_label: "address"

This relabeling rule joins host and port label values with : separator and stores the result at target_label .

Sometimes it is necessary to construct a label from parts of existing labels. For example, the url label with http://hostname/path value must be constructed from address and path labels, where address contains hostname:port . This can be done with the following relabeling rule:

- source_labels: [address, path]
  regex: "([^:]+):[^;]+;(.*)"
  replacement: "http://$1/$2"
  target_label: "url"

By default source_labels are joined with ; separator before applying regex to the resulting string. So the resulting string will contain address:port;path value. The given regex matches this string and extracts address and path parts into $1 and $2. Then these values are used for constructing replacement string in the form http://<address>/<path> . Then the resulting string is written into target_label.

Chaining relabeling rules

Relabeling rules can be chained. For example, the following rules add {foo="bar"} label and remove port from instance label:

- replacement: "bar"
  target_label: "foo"
- source_labels: [instance]
  regex: "([^:]+):.*"
  replacement: "$1"
  target_label: "instance"

Arbitrary number of relabeling rules can be chained.

Conditional relabeling

Sometimes it may be needed to apply relabeling only for a subset of metrics. In this case the if option can be added to relabeling rule. For example, the following rule adds {foo="bar"} label only to metrics matching metric{label=~"x|y"} series selector:

- if: 'metric{label=~"x|y"}'
  target_label: "foo"
  replacement: "bar"

The if option can simplify traditional relabeling rules. For example, the following rule drops metrics, which match foo{bar="baz"} series selector:

- if: 'foo{bar="baz"}'
  action: drop

This is equivalent to the following less clear traditional relabeling rule:

- action: drop
  source_labels: [__name__, bar]
  regex: "foo;baz"

Unfortunately the if option isn’t supported by Prometheus yet — it is supported by VictoriaMetrics only :(

Conclusion

Relabeling in Prometheus and VictoriaMetrics is quite powerful. It allows performing arbitrary transformations on metric names, label names and label values. Relabeling can be applied at the following stages:

• at scrape_configs->relabel_configs section for transforming per-scrape target labels and dropping unneeded scrape targets
• at scrape_config->metric_relabel_configs section for transforming and filtering per-metric labels before storing them in persistent storage
• at remote_write->write_relabel_configs section for transforming and filtering per-metric labels before writing them to remote storage

See also docs about relabeling in vmagent. It supports relabeling for any collected metrics, including data obtained via various supported data ingestion protocols such as Influx line protocol, Graphite protocol, OpenTSDB protocol, CSV, etc. See these docs for more details on supported data ingestion protocol in VictoriaMetrics and vmagent.

P.S. VictoriaMetrics and vmagent provide many useful features additionally to relabeling. Visit VictoriaMetrics site in order to learn more.