Get Down(sampled) With Prometheus Recording Rules in Promscale

Get Down(sampled) With Prometheus Recording Rules in Promscale
⚠️
While part of this content may still be up to date, we regret to inform you that we decided to sunset Promscale in February 2023. Read our reasons why and other FAQs.

This is KubeCon Europe week, and at Timescale, we’re celebrating it the way we know best: with a new edition of #AlwaysBeLaunching! 🐯 🎊 During this week, we are releasing new features and content every day, all focused on Promscale and observability. Do not miss any of it by following our hashtag #PromscaleKubecon on Twitter—and if you’re at KubeCon, come visit us at booth G3! We have awesome demos (and swag) to give you!

One more day, one more launch! Yesterday, we announced the support for Prometheus Alerting Rules in Promscale—and today, it’s the Recording Rules’ turn! 🔥

In Prometheus, Recording Rules allow you to periodically write the output of PromQL expressions that are frequently used or computationally expensive as a new time series, so you can conveniently query them later while getting faster response times.

Recording Rules are very useful as a downsampling mechanism, as you can use them to roll up metrics into larger time periods. By directly configuring Recording Rules in Promscale, you can define a retention policy that actually drops the source data after a predefined amount of time, which is not possible to do directly with a single Prometheus instance which has a single retention time for all metrics.

But Recording Rules are not the only way to downsample data in Promscale. Since Promscale is built on top of PostgreSQL and TimescaleDB, you can both use Prometheus’ Recording Rules written in PromQL and TimescaleDB’s continuous aggregates written in SQL for downsampling. This gives you maximum flexibility to choose the method and query language that better adapts to the particular metric you’d like to downsample. For more information on the differences between both methods and when to use one or the other, check out this blog post we published on the topic.

To learn how you can configure recording rules in Promscale, keep reading. And if you want to try out Promscale, you can install it for free here.

✨ Promscale is the unified observability backend for metrics and traces built on PostgreSQL and TimescaleDB. It brings you observability powered by SQL without losing native support for Prometheus and PromQL; you can store your Prometheus metrics and your OpenTelemetry traces in Promscale, using it together with Grafana and Jaeger for an integrated observability experience. If you’re new to Promscale, check out our website and documentation to learn more.

Configuring Recording Rules in Promscale

Prometheus Recording Rules allow the creation of new metrics from existing metrics using PromQL, usually because we know we are going to use them frequently or they run too slowly to generate at view time. For example, if we know that we are always going to sum our requests by HTTP response code, it might be worth looking at a Recording Rule, especially if we have a very large number of instances.

A basic recording rule is expressed as a record (which defines the name of the new time series) and an expression (which defines the query to run). Promscale supports the full PromQL language, so any rule which Prometheus can evaluate can also be used in Promscale on similar metric data to create new time series with Recording Rules.

Recording Rules are configured in Promscale similarly to in Prometheus—using a YAML file that points at YAML files containing recording rules to load. In fact, both file formats are identical, so if you have a working Prometheus configuration that includes a rule_files block and some recording rules, you can point Promscale at the config, and things will work.


So what does that config look like? A basic example of the config needed would be a prometheus.yml with the following content:

# Rules and alerts are read from the specified file(s)
rule_files:
  - rules.yml

Rules will be read from the rules.yml which contains a list of PromQL rules (or alerts) to load and evaluate. An example of the file with a recording rule that pre-calculates the rate of CPU usage would look like this:

groups:
- name: rules
  rules:
  - record: instance_cpu:node_cpu_seconds_not_idle:rate5m
    expr: >
      sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) 
      without (mode,cpu)

Once we have the config files, Promscale can be started with either the metrics.rules.prometheus-config or the PROMSCALE_METRICS_RULES_PROMETHEUS_CONFIG environment variable pointing at the prometheus.yml file.

Check Out an Example

If you’d like to give this a go, then you can use the docker-compose file, which is located in the https://github.com/timescale/promscale repository, to see a recording rule working.

If you execute the following commands from a shell:

git clone https://github.com/timescale/promscale
cd promscale/docker-compose
docker-compose up

Then you should see the stack come up. After waiting for it to stabilize you can connect to the TimescaleDB database and see the data from the recording rule with the following command:

docker exec -it docker-compose-db-1 psql -c \
  'SELECT * FROM "instance_cpu:node_cpu_seconds_not_idle:rate5m"   
  LIMIT 10'

You should see a few rows of metric data. (If you don’t, you may need to wait a little longer for the rule to evaluate.)

Wrap Up

Promscale now natively supports both alerting rules and recording rules, giving users the option to not run a full Prometheus instance for collecting metrics (saving valuable machine resources) while still maintaining access to these great features.

If you have any questions about this new functionality, please reach out to us—we’ll be more than happy to help! You can find us in our Community Slack (make sure to join the #promscale channel). And for hands-on technical questions, you can also post in the Timescale Forum.

#AlwaysBeLaunching!

Ingest and query in milliseconds, even at terabyte scale.
This post was written by
4 min read
Observability
Contributors

Related posts