Multi-tenancy support for Prometheus in Promscale

Promscale supports the ingestion of metrics from different Prometheus tenants.

How it works

Each Prometheus instance needs to be configured to indicate which tenant it belongs to when setting up remote_write . This will ensure that the correct tenant is attributed when sending data to the Promscale Connector.

Once configured, the Promscale Connector ensures each metric is decorated with the respective __tenant__ label, so that the data can be differentiated by the tenant name. This allows you to write queries for specific tenants, or across multiple tenants.

To configure permissions, you set up an additional Promscale Connector, and configure it to only allow query access to a specific tenant or group of tenants - only allowing those users to query data via that Promscale Connector. This is typically achieved by configuring a Prometheus data source for the Promscale Connector in Grafana, and then setting up the appropriate data source permissions.

How to configure multi-tenancy

First, you have to configure Prometheus to send tenant information to Promscale, so that Promscale can identify which tenant the data originated from. There are two ways to do this, both of them being done through the Prometheus configuration file:

  • Pass a __tenant__ label with all metrics (recommended)

In Prometheus, you can leverage external labels for this. Prometheus will automatically add the __tenant__ label to all metrics before they’re sent to Promscale.

global:
 scrape_interval:    5s
 evaluation_interval: 30s
 external_labels:
   __tenant__: tenant-A
  • Use the TENANT HTTP header

The Prometheus configuration file allows you to set any number of HTTP headers to be sent with every remote_write request to Promscale.

remote_write:
-  url: http://localhost:9201/write
   headers:
     TENANT: team-1

Once set up, the Promscale Connector will retrieve the value of the TENANT HTTP header. If that tenant is authorized in that Promscale Connector, Promscale will ingest and decorate all the metrics in the remote_write request by appending a tenant label using the value of the TENANT header.

Once Prometheus is configured, you can enable multi-tenancy in Promscale by setting the PROMSCALE_MULTI_TENANCY=true environment variable when starting the Promscale Connector, or by passing the -multi-tenancy parameter. With this, Promscale will accept data from all tenants, both for write and read, and will add the corresponding __tenant__ label to incoming metrics.

If you want Promscale to allow ingest and query for data only from specific tenants, pass those tenant names separated by commas via the -multi-tenancy-valid-tenants parameter or the PROMSCALE_MULTI_TENANCY_VALID_TENANTS environment variable.

For example, if SomeCompany wants to allow Promscale to ingest and query data only for development teams 1 and 2, they’d set parameters like so:

-multi-tenancy-valid-tenants=team-1,team-2

With that setting, only data corresponding to team-1 or team-2 will be available, and Promscale will ignore and report all other data as unauthorized.

By default, the -multi-tenancy-valid-tenants has the value allow-all, allowing all incoming tenants to be ingested and queried.

When multi-tenancy is enabled, Promscale drops all data from a Prometheus instance that isn’t configured to send tenant information. You instruct Promscale to ingest the data by passing the -multi-tenancy-allow-non-tenants parameter or the PROMSCALE_MULTI_TENANCY_ALLOW_NON_TENANTS=true environment variable when launching the Promscale Connector.

This information was originally published in this blogpost. Check it out for further insights on multi-tenancy!