Invalidation log size

I’ve noticed that our invalidation log is high. It is 59 GB in size, almost the size of our largest hypertable at 69 GB after compression.

I think this might have to do with how we ingest data. We’re receiving data from clients and shipping it through an AWS SQS standard (no FIFO guarantees), and a single worker writing to Timescale. Either there’s a large variability in ordering, or that the timestamps we are sending out lag enough to trigger writes into the invalidation log.

So I got some questions:

  • Can the invalidation log be pruned? Are there log entries that can be deleted after being processed for materialization? I’m attempting a migration and would like to reduce the data we have to dump and restore.
  • How can I find out if we’re writing excessively to the invalidation log too high because we have too many things out of order, and/or the materialization threshold is set too low?
  • Can materialization threshold be modified and tuned?

:wave: @hosh,

What version of TimescaleDB are you using?