SageMaker cuts LLM latency 77% with prefix routing

Amazon SageMaker Inference now offers prefix-aware routing, a strategy that sends requests sharing the same prompt prefix to the same instance, allowing KV cache reuse across repeated requests. This solves a core inefficiency where multi-instance deployments prevented prefix caching from working effectively. Benchmarks on Llama 3.1 70B across seven instances showed P50 time-to-first-token reduced by up to 77 percent, throughput increased by 16 percent, and KV cache hit rates jumping from 25 to over 80 percent. Routing overhead adds only 1.3 to 1.9 milliseconds per request, and built-in overload protection prevents any single instance from becoming a bottleneck.