Of the three observability data types supported by OpenTelemetry (metrics, logs, and traces), the latter presents the most potential. Tracing exposes the inner-workings of requests processing in modern, cloud-native architectures.
Tools typically visualize a single trace, showing how an individual request traversed your entire system. This approach helps with distributed debugging and analysis. However, looking at a single trace is of limited value.
But what if you stored tracing data in a SQL database? You could ask global questions about your system and uncover answers aggregated from thousands of traces at once. You could discover in real-time upstream and downstream dependencies between services or components. You could find slow execution paths, where the error rate spiked since the last deployment, or where the request rate suddenly dropped - and not just that it occurred but pinpoint the cause. Thus, tracing analysis is a proactive tool to help you spot issues before your customers do. In this workshop, you will learn how to do precisely that.
Concretely, we will ingest OpenTelemetry traces into a PostgreSQL database, and we’ll learn how to build powerful custom dashboards in Grafana using SQL that will immediately give you a deep level of insight into your distributed systems, helping you make the most out of your tracing data.
First, we will explore a lightweight open-source system designed to demonstrate the power of tracing and SQL that anyone can easily download and run.
Then, we will use this demo environment to explore how to search and visualize traces using Jageer and Grafana, and as the last step, we will learn how to create a new Grafana dashboard to visualize throughput, response time and a recursive query for analyzing downstream dependencies through tracing data.
Github repo: tsdb.co/opentelemetry-demo