Vast tiers storage to manage AI agent memory

AI agents running long sessions consume significant GPU memory, with half-a-million-token sessions occupying up to one-tenth of a GPU's capacity. Vast CTO Alon Horev says the company addresses this by tiering memory across GPU, CPU, and persistent storage, using Nvidia's Dynamo software to orchestrate KV cache offloading and session migration across machines. This approach reduces repeat recalculation and improves scheduling across GPU fleets. Horev also noted that agent conversation data has value for fine-tuning and training, and that Vast has launched a confidential computing service for sensitive enterprise workloads.