Introducing NVIDIA BlueField-4-Powered Inference Context Memory Storage Platform for the Next Frontier of AI

Organizations face recent scalability challenges as models evolve from easy chatbots to complex, multiturn agentic workflows. With foundation models reaching trillions of parameters and context windows spanning thousands and thousands of tokens, the three AI scaling laws (pretraining, post-training, and test-time scaling) are driving a surge in compute-intensive reasoning. Agents aren’t any longer stateless chatbots and depend upon long‑term memory of conversations, tools, and intermediate results, shared across services and revisited over time.

In transformer-based models, that long‑term memory is realized as inference context, also referred to as KV cache. This preserves inference-time context so the model doesn’t recompute history for each recent token. As sequence lengths increase, the KV cache grows linearly, forcing it to persist across longer sessions and be shared across inference services. 

This evolution positions KV cache as a novel class of AI‑native data defined by a selected duality: it’s critical for performance yet inherently ephemeral. In agentic systems, KV cache effectively becomes the model’s long‑term memory, reused and prolonged across many steps reasonably than discarded after a single-prompt response. 

Unlike immutable enterprise records, inference context is derived and recomputable, demanding a storage architecture that prioritizes power and price efficiency in addition to speed and scale, over traditional data durability. In modern AI infrastructure, which means every megawatt of power is ultimately judged by what number of useful tokens it will possibly deliver. 

Meeting these requirements stretches today’s memory and storage tiers to their limits. For this reason organizations are rethinking how context is placed across GPU memory, host memory, and shared storage.

To know the gap, it’s helpful to have a look at how inference context currently moves across the G1–G4 hierarchy (Figure 1). AI infrastructure teams use orchestration frameworks, equivalent to NVIDIA Dynamo, to assist manage this context across these storage tiers: 

• G1 (GPU HBM) for decent, latency‑critical KV utilized in lively generation 
• G2 (system RAM) for staging and buffering KV off HBM
• G3 (local SSDs) for warm KV that’s reused over shorter timescales 
• G4 (shared storage) for cold artifacts, history, and results that have to be durable but aren’t on the immediate critical path 

G1 is optimized for access speed while G3 and G4 are optimized for durability. As context grows, KV cache quickly exhausts local storage capability (G1-G3), while pushing it all the way down to enterprise storage (G4), which introduces unacceptable overheads and drives up each cost and power consumption. 

Figure 1 illustrates this tradeoff, showing how KV cache usage becomes increasingly expensive because it moves farther from the GPU across the memory and storage hierarchy.

At the highest of the hierarchy, GPU HBM (G1) delivers nanosecond-scale access and the very best efficiency, making it ideal for lively KV cache used directly in token generation. As context grows beyond the physical limits of HBM, KV cache spills into system DRAM (G2) and native/rack-attached storage (G3), where access latency increases and per-token energy and price begin to rise. While these tiers extend effective capability, each additional hop introduces overhead that reduces overall efficiency.

At the underside of the hierarchy, shared object and file storage (G4) provides durability and capability, but at millisecond-level latency and the bottom efficiency for inference. While suitable for cold or shared artifacts, pushing lively or regularly reused KV cache into this tier drives up power consumption, and directly limits cost-efficient AI scaling.

The important thing takeaway is that latency and efficiency are tightly coupled: as inference context moves away from the GPU, access latency increases, energy use and price per token rise, and overall efficiency declines. This growing gap between performance-optimized memory and capacity-optimized storage is what forces AI infrastructure teams to rethink how growing KV cache context is placed, managed, and scaled across the system.

AI factories need a complementary, purpose‑built context layer that treats KV cache as its own AI‑native data class reasonably than forcing it into either scarce HBM or general‑purpose enterprise storage.

Introducing the NVIDIA Inference Context Memory Storage platform 

The NVIDIA Inference Context Memory Storage platform is a totally integrated storage infrastructure. It uses the NVIDIA BlueField-4 data processor to create a purpose-built context memory tier operating on the pod level to bridge the gap between high-speed GPU memory and scalable shared storage. This accelerates KV cache data access and high-speed sharing across nodes inside the pod to boost performance and optimize power consumption for the growing demands of large-context inference.

The platform establishes a brand new G3.5 layer, an Ethernet-attached flash tier optimized specifically for KV cache. This tier acts because the agentic long‑term memory of the AI infrastructure pod that’s large enough to carry shared, evolving context for a lot of agents concurrently, but additionally close enough for the context to be pre‑staged regularly back into GPU and host memory without stalling decode. 

It provides petabytes of shared capability per GPU pod, allowing long‑context workloads to retain history after eviction from HBM and DRAM. The history is stored in a lower‑power, flash‑based tier that extends the GPU and host memory hierarchy. The G3.5 tier delivers massive aggregate bandwidth with higher efficiency than classic shared storage. This transforms KV cache right into a shared, high‑bandwidth resource that orchestrators can coordinate across agents and services without rematerializing it independently on each node.

With a big portion of latency-sensitive, ephemeral KV cache now served from the G3.5 tier, durable G4 object and file storage could be reserved for what truly must persist over time. This includes inactive multiturn KV state, query history, logs, and other artifacts of multiturn inference which may be recalled in later sessions.

This reduces capability and bandwidth pressure on G4 while still preserving application-level history where it matters. As inference scale increases, G1–G3 KV capability grows with the variety of GPUs but stays too small to cover all KV needs. ICMS fills this missing KV capability between G1–G3 and G4.   

Inference frameworks like NVIDIA Dynamo use their KV block managers along with NVIDIA Inference Transfer Library (NIXL) to orchestrate how inference context moves between memory and storage tiers, using ICMS because the context memory layer for KV cache. KV managers in these frameworks prestage KV blocks, bringing them from ICMS into G2 or G1 memory ahead of the decode phase. 

This reliable prestaging, backed by the upper bandwidth and higher power efficiency of ICMS in comparison with traditional storage, is designed to reduce stalls and reduce idle time, enabling as much as 5x higher sustained TPS for long-context and agentic workloads. When combined with the NVIDIA BlueField-4 processor running the KV I/O plane, the system efficiently terminates NVMe-oF and object/RDMA protocols.

Figure 2 shows how ICMS suits into the NVIDIA Rubin platform and AI factory stack. 

On the inference layer, NVIDIA Dynamo and NIXL manage prefill, decode, and KV cache while coordinating access to shared context. Under that, a topology-aware orchestration layer using NVIDIA Grove places workloads across racks with awareness of KV locality so workloads can proceed to reuse context at the same time as they move between nodes. 

On the compute node level, KV tiering spans GPU HBM, host memory, local SSDs, ICMS, and network storage, providing orchestrators with a continuum of capability and latency targets for putting context. Tying all of it together, Spectrum-X Ethernet links Rubin compute nodes with BlueField-4 ICMS goal nodes, providing consistently low latency and efficient networking that integrates flash-backed context memory into the identical AI-optimized fabric that serves training and inference.

Powering the NVIDIA Inference Context Memory Storage platform

NVIDIA BlueField-4 powers ICMS with 800 Gb/s connectivity, 64-core NVIDIA Grace CPU, and high-bandwidth LPDDR memory. Its dedicated hardware acceleration engines deliver line‑rate encryption and CRC data protection at as much as 800 Gb/s. 

These crypto and integrity accelerators are designed for use as a part of the KV pipeline, securing and validating KV flows without adding host CPU overhead. By leveraging standard NVMe and NVMe-oF transports, including NVMe KV extensions, ICMS maintains interoperability with standard storage infrastructure while delivering the specialized performance required for KV cache. 

The architecture uses BlueField‑4 to speed up KV I/O and control plane operations, across DPUs on the Rubin compute nodes and controllers in ICMS flash enclosures, reducing reliance on the host CPU and minimizing serialization and host memory copies. Moreover, Spectrum‑X Ethernet provides the AI‑optimized RDMA fabric that links ICMS flash enclosures and GPU nodes with predictable, low‑latency, high‑bandwidth connectivity.

Moreover, the NVIDIA DOCA framework introduces a KV communication and storage layer that treats context cache as a first-class resource for KV management, sharing, and placement, leveraging the unique properties of KV blocks and inferencing patterns. DOCA interfaces inference frameworks, with BlueField-4 transferring the KV cache efficiently to and from the underlying flash media. 

This stateless and scalable approach aligns with AI-native KV cache strategies and leverages NIXL and Dynamo for advanced sharing across AI nodes and improved inference performance. The DOCA framework supports open interfaces for broader orchestration, providing flexibility to storage partners to expand their inference solutions to cover the G3.5 context tier.

Spectrum-X Ethernet serves because the high-performance network fabric for RDMA-based access to AI-native KV cache, enabling efficient data sharing and retrieval for the NVIDIA Inference Context Memory Storage platform. Spectrum-X Ethernet is purpose-built for AI, delivering predictable, low-latency, high-bandwidth connectivity at scale. It achieves this through advanced congestion control, adaptive routing, and optimized lossless RoCE, which minimizes jitter, tail latency, and packet loss under heavy load. 

With very high effective bandwidth, deep telemetry, and hardware-assisted performance isolation, Spectrum-X Ethernet enables consistent, repeatable performance in large, multitenant AI fabrics while remaining fully standards-based and interoperable with open networking software. Spectrum-X Ethernet enables ICMS to scale with consistent high performance, maximizing throughput and responsiveness for multiturn, agentic inference workloads.

Delivering power‑efficient, high-throughput KV cache storage

Power availability is the first constraint for scaling AI factories, making energy efficiency a defining metric for gigascale inference. Traditional, general-purpose storage stacks sacrifice this efficiency because they run on x86‑based controllers and expend significant energy on features like metadata management, replication, and background consistency checks which can be unnecessary for ephemeral, reconstructable KV data.

KV cache fundamentally differs from enterprise data: it’s transient, derived, and recomputable if lost. As inference context, it doesn’t require the sturdiness, redundancy, or extensive data protection mechanisms designed for long-lived records. Applying these heavy storage services to KV cache introduces unnecessary overhead, increasing latency and power consumption while degrading inference efficiency. By recognizing KV cache as a definite, AI-native data class, ICMS eliminates this excess overhead, enabling as much as 5x improvements in power efficiency in comparison with general-purpose storage approaches.

This efficiency extends beyond the storage tier to the compute fabric itself. By reliably prestaging context and reducing or avoiding decoder stalls, ICMS prevents GPUs from wasting energy on idle cycles or redundant recomputation of history, which ends up in as much as 5x higher TPS. This approach ensures that power is directed toward lively reasoning reasonably than infrastructure overhead, maximizing effective tokens‑per‑watt for your entire AI pod.

Enabling gigascale agentic AI with higher performance and TCO

The BlueField‑4–powered ICMS provides AI‑native organizations with a brand new option to scale agentic AI: a pod‑level context tier that extends effective GPU memory and turns KV cache right into a shared high‑bandwidth, long‑term memory resource across NVIDIA Rubin pods. By offloading KV movement and treating context as a reusable, nondurable data class, ICMS reduces recomputation and decode stalls, translating higher tokens‑per‑second directly into more queries served, more agents running concurrently, and shorter tail latencies at scale.

Together, these gains improve total cost of ownership (TCO) by enabling teams to suit more usable AI capability into the identical rack, row, or data center, extend the lifetime of existing facilities, and plan future expansions around GPU capability as a substitute of storage overhead.

To learn more concerning the NVIDIA BlueField-4-powered Inference Context Memory Storage platform, see the press release and the NVIDIA BlueField-4 datasheet. 

Watch NVIDIA Live at CES 2026 with CEO Jensen Huang and explore related sessions.