CNCF-hosted Co-located Events Europe 2025

At Uber, we give service owners the autonomy to build and deploy what they think is right to get the job done. However, this comes with the challenge of applying a consistent baseline of encryption and authorization across dozens of languages, frameworks, and off-the-shelf software. To solve this problem, we needed to re-orient from targeting individual applications and frameworks to a foundational solution that worked regardless of application type.

This talk will cover Uber's journey from having targeted encryption, authentication, and authorization deployed internally to 100% coverage in the span of 2 years by transparently encrypting, authenticating, and authorizing billions of connections. We will cover how Envoy enabled and accelerated this transformation as well as our operational learnings from development, deployment, and our first year with Uber fully onboarded.

Envoy has great support for rate limiting, but as you scale up you’ll undoubtedly face serious cost and operational challenges.

Envoy’s global rate-limiting makes decisions on the request hot path, leading to an unavoidable latency increase, and a linear increase in requests to your rate limiting cluster. Spotify had noticed that the cost of running the rate limiting cluster was roughly equivalent to running the Envoys themselves!

Spotify addressed these scaling challenges by writing a new distributed Envoy rate limiter, and did so as an Envoy filter written in golang, and a backing Java service.

Do you have similar challenges? Learn about why Spotify chose to develop a new rate limiting approach, how the approach scales to meet Spotify’s needs, and the experience of writing a non-trivial filter in golang.

Deploying Large Language Models (LLMs) efficiently in production environments presents unique challenges. This talk explores how Envoy proxy, a popular open-source edge and service proxy, has been enhanced to address these challenges. We'll delve into new features and techniques in Envoy that optimize LLM serving, improve performance, and simplify integration into Kubernetes-native architectures.

Key Takeaways:
Understand the specific challenges of deploying and scaling LLMs in production.
Learn how Envoy's latest features address these challenges, including:
** Advanced load balancing for LLM inference: Discuss how Envoy can intelligently route requests to optimize resource utilization and minimize latency.
** LLM Model Awareness: Explain how Envoy can be instrumented for compatibility with popular LLM serving specifications such as OpenAI API specifications.
** Security considerations for LLMs: How you can attach AI Safety frameworks in the Envoy proxy dataplane .

For years, the Envoy community has eagerly awaited support for dynamic modules as an extension mechanism – a feature that remained unaddressed until recently. A dynamic module is a shared library that can be loaded by Envoy at runtime, offering an alternative to existing extension mechanisms like Lua, External Processing, or Wasm.

Our early benchmark results in a production environment demonstrate that dynamic modules perform almost identically to native C++ extensions, which was nearly impossible before. Developing native C++ extensions requires rebuilding the Envoy binary and entails significant maintenance. In contrast, dynamic modules can be developed in almost any programming language, hence greatly benefits vendors and end users who require performance-sensitive custom business logic with the minimum maintenance.

In this talk, as the maintainer of dynamic module support in Envoy, I will provide an overview of this feature, share recent updates, and discuss future developments.

Recent changes to Envoy provide a way to utilize load reports obtained from backends via Open Request Cost Aggregation (ORCA).

The initial implementation supports direct (inline) load reports from backends in the form of response headers and used in two different ways.
1. Load Reports are provided to the xDS control plane server via xDS LRS API.
2. Load reports are used by a new Client Side Weighted Round Robin load balancing policy to dynamically calculate host weights on the client side. Inline reporting enables sub second load balancing reaction times, a critical requirement for customers with coordinated and spiky traffic workloads.

Using these load reports, Envoy proxies are able to implement load balancing policies that vary endpoint load balancing weights according to backend load reports.
This talk discusses some high level changes that ORCA load reporting introduces as well as potential enhancements down the line.

In this session attendees will get an in depth overview of how Envoy manages memory allocations with the help of Google tcmalloc allocator. We will also go over on demand memory releasing techniques in Envoy and shed light on why they don't operate in a fully predictable way. Lastly we will reveal the future plans of supporting more allocator types that can be leveraged for better memory management for certain deployment types.