Get Down(sampled) With Prometheus Recording Rules in Promscale

One more day, one more launch! Yesterday, we announced the support for Prometheus Alerting Rules in Promscale—and today, it’s the Recording Rules’ turn! 🔥

In Prometheus, Recording Rules allow you to periodically write the output of PromQL expressions that are frequently used or computationally expensive as a new time series, so you can conveniently query them later while getting faster response times.

Recording Rules are very useful as a downsampling mechanism, as you can use them to roll up metrics into larger time periods. By directly configuring Recording Rules in Promscale, you can define a retention policy that actually drops the source data after a predefined amount of time, which is not possible to do directly with a single Prometheus instance which has a single retention time for all metrics.

But Recording Rules are not the only way to downsample data in Promscale. Since Promscale is built on top of PostgreSQL and TimescaleDB, you can both use Prometheus’ Recording Rules written in PromQL and TimescaleDB’s continuous aggregates written in SQL for downsampling. This gives you maximum flexibility to choose the method and query language that better adapts to the particular metric you’d like to downsample. For more information on the differences between both methods and when to use one or the other, check out this blog post we published on the topic.

To learn how you can configure recording rules in Promscale, keep reading. And if you want to try out Promscale, you can install it for free here.

Configuring Recording Rules in Promscale

Prometheus Recording Rules allow the creation of new metrics from existing metrics using PromQL, usually because we know we are going to use them frequently or they run too slowly to generate at view time. For example, if we know that we are always going to sum our requests by HTTP response code, it might be worth looking at a Recording Rule, especially if we have a very large number of instances.

A basic recording rule is expressed as a record (which defines the name of the new time series) and an expression (which defines the query to run). Promscale supports the full PromQL language, so any rule which Prometheus can evaluate can also be used in Promscale on similar metric data to create new time series with Recording Rules.

Recording Rules are configured in Promscale similarly to in Prometheus—using a YAML file that points at YAML files containing recording rules to load. In fact, both file formats are identical, so if you have a working Prometheus configuration that includes a rule_files block and some recording rules, you can point Promscale at the config, and things will work.

So what does that config look like? A basic example of the config needed would be a prometheus.yml with the following content:

# Rules and alerts are read from the specified file(s)
rule_files:
  - rules.yml

Rules will be read from the rules.yml which contains a list of PromQL rules (or alerts) to load and evaluate. An example of the file with a recording rule that pre-calculates the rate of CPU usage would look like this:

groups:
- name: rules
  rules:
  - record: instance_cpu:node_cpu_seconds_not_idle:rate5m
    expr: >
      sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) 
      without (mode,cpu)

Once we have the config files, Promscale can be started with either the metrics.rules.prometheus-config or the PROMSCALE_METRICS_RULES_PROMETHEUS_CONFIG environment variable pointing at the prometheus.yml file.

Check Out an Example

If you’d like to give this a go, then you can use the docker-compose file, which is located in the https://github.com/timescale/promscale repository, to see a recording rule working.

If you execute the following commands from a shell:

git clone https://github.com/timescale/promscale
cd promscale/docker-compose
docker-compose up

Then you should see the stack come up. After waiting for it to stabilize you can connect to the TimescaleDB database and see the data from the recording rule with the following command:

docker exec -it docker-compose-db-1 psql -c \
  'SELECT * FROM "instance_cpu:node_cpu_seconds_not_idle:rate5m"   
  LIMIT 10'

You should see a few rows of metric data. (If you don’t, you may need to wait a little longer for the rule to evaluate.)

Wrap Up

Promscale now natively supports both alerting rules and recording rules, giving users the option to not run a full Prometheus instance for collecting metrics (saving valuable machine resources) while still maintaining access to these great features.

If you have any questions about this new functionality, please reach out to us—we’ll be more than happy to help! You can find us in our Community Slack (make sure to join the #promscale channel). And for hands-on technical questions, you can also post in the Timescale Forum.

#AlwaysBeLaunching!