CNCF-hosted Co-located Events Europe 2025

Deploying Large Language Models (LLMs) efficiently in production environments presents unique challenges. This talk explores how Envoy proxy, a popular open-source edge and service proxy, has been enhanced to address these challenges. We'll delve into new features and techniques in Envoy that optimize LLM serving, improve performance, and simplify integration into Kubernetes-native architectures.

Key Takeaways:
Understand the specific challenges of deploying and scaling LLMs in production.
Learn how Envoy's latest features address these challenges, including:
** Advanced load balancing for LLM inference: Discuss how Envoy can intelligently route requests to optimize resource utilization and minimize latency.
** LLM Model Awareness: Explain how Envoy can be instrumented for compatibility with popular LLM serving specifications such as OpenAI API specifications.
** Security considerations for LLMs: How you can attach AI Safety frameworks in the Envoy proxy dataplane .