Open Source Summit + Embedded Linux Conference North America 2026

LMCache supports tiered KV caching with CPU memory offloading, extending inference beyond GPU memory limits. But what happens when local CPU memory isn't enough? This session introduces the next tier: offloading KV cache to Amazon SageMaker HyperPod managed storage, expanding cache capacity for large-scale LLM inference.

We'll cover the technical design of the SageMaker HyperPod connector contribution to LMCache. Hot entries stay in GPU memory, warm entries spill to CPU memory, and cold entries persist to HyperPod's managed storage. This three-tier architecture lets organizations cache far more context than local resources allow, reducing redundant computation for repeated prompts and long-context scenarios.

The session demonstrates the integration in action, showing cache hit rates, latency across tiers, and how the connector handles transitions between local and remote storage. We'll discuss key engineering decisions, including async prefetching and failure handling.

Attendees will leave with practical knowledge of how managed cloud storage can extend open source caching frameworks for LLM inference infrastructure.