SageMaker Gateway Cuts AI Latency 82%

Amazon has launched SageMaker HyperPod Inference Gateway, a Kubernetes-native routing system that reduces first-token latency by up to 82% for large language model deployments. It installs as a single EKS managed addon with no application code changes required. Unlike standard round-robin load balancers, the gateway uses real-time GPU metrics including KV cache utilization, queue depth, and LoRA adapter residency to route each request to the optimal pod. A second tier offering cross-cluster failover and global traffic shaping is coming soon.