Hi all, I am trying to get myself up to speed on the best ways to handle logging within Kubernetes. I have seen various stacks that are commonplace, many of them using Elasticsearch as the persistence store, and I wondered what Elastic can do that Postgres can’t. I have also read that Elasticsearch is quite a resource hog so I am reluctant to put another heavyweight store into the cluster if it doesn’t come with some heavyweight benefits.
I then discovered this CNCF video which shows some (early days?) experiments with capturing, tagging and formatting logs using Fluent Bit before storing and querying in Postgres. It looked like some of the queries he is attempting could definitely benefit from the Timescale extension!
So I thought it was worth starting a general discussion here to understand:
- what work, if any, has been done to incorporate logging into Timescale or the tobs stack?
- is logging on a roadmap? (I can imagine that, as a feature, it doesn’t fit neatly into either TimescaleDB or Promscale… perhaps a 3rd product?)
- what are the friction points; what can Elasticsearch or some other store do which would be hard/impossible to achieve in Postgres?
I have read that logging is one of the 3 pillars of observability. And with Fluent Bit’s ability to tag and structure the raw log data before presenting it to Postgres, could this be a crucial missing element to complete the tobs stack?