CNCF-hosted Co-located Events Europe 2025

Learn how, in our quest for observability, we accidentally sent 300 million time series to our stack, repeatedly crashed our OTel agents due to cascading failures, and generally made our Observability SRE's lives miserable. In this talk, we'll share the key lessons learned from our missteps, including best practices for scaling observability in complex systems, avoiding common pitfalls, and building resilient monitoring pipelines (using OTel, VictoriaMetrics, Loki and Tempo). Join us to understand how a combination of over-ambitious instrumentation and lack of foresight can lead to chaos — and how to prevent it in your own organization. By the end, you'll have actionable insights to optimize your observability strategy without breaking the system (or the team).