CNCF-hosted Co-located Events Europe 2025

Enterprises have critical operational and product data distributed in a variety of sources that includes public documentation, internal wikis and various ticketing systems. With the rising capabilities of Large Language Models, enterprises & SMBs are looking to LLMs to help achieve valuable insights from these diverse data sources accessible via natural language querying.

In this talk, we illustrate how we can build a self-hosted RAG system to answer questions across these data sources powered by LLMs hosted on a scalable and observable Kubernetes cluster. Auto-scaling and observability are incorporated into our Question-Answering-in-a-Box stack both at the application and infrastructure level to ensure that GenAI app developers can start with pilot deployments and then systematically move through an Iterate-and-Improve cycle to eventually deploy their stack in production in a cost-effective fashion.