SK海力士在OCP峰会展示MARS内存池化架构与HBF新内存层，HBF硬件要到2028年才送样

SK Hynix has presented the Memory-Centric AI Rack System (MARS) at the Open Compute Project (OCP) Global Summit. The architecture separates memory from individual servers and pools it at the rack level over Compute Express Link (CXL) fabrics. The company says this addresses the capacity and cost limits that are constraining the growth of large AI models.

Hoshik Kim, Senior Vice President and Fellow of Memory Systems Research at SK Hynix America, gave the presentation. He argued that today's processor-centric designs, where memory is fixed to each CPU or GPU, can't keep pace with large language models (LLMs) and multi-turn agentic AI workloads.
"Unlocking the next level of AI performance requires fundamental changes in computer architecture... We need to rethink how data is stored, accessed, processed, and shared," Kim said.

Credit: DIGITIMES
SK hynix described several bottlenecks in current infrastructure. Adding High Bandwidth Memory (HBM) to GPUs is limited by physical space and cost, and buying extra GPUs just for memory capacity is not economical. Memory on busy nodes often runs out while memory on nearby nodes in the same cluster sits unused.
Adding memory usually means buying whole new servers, which pushes operators to over-provision costly GPU clusters. Moving data such as key-value (KV) caches between nodes over Remote Direct Memory Access (RDMA) networks also adds repeated data copying and unpredictable tail latency.
MARS turns DRAM into a shared, rack-scale pool. Capacity can be allocated, scaled, and reclaimed separately from CPU and GPU compute. CXL interconnects link compute servers and memory trays inside the rack, giving direct memory access without network overhead at the application level.
Multiple GPUs and NPUs can read one shared pool holding LLMs and context KV caches. In pipelined workloads, one accelerator can write its output to shared memory, and the next can read it directly, with no explicit transfer across an RDMA network.
SK Hynix shared performance data from three deployment scenarios. In multi-turn agentic AI workloads, offloading growing KV caches to the shared MARS pool improved time-to-first-token (TTFT) by 3.5 to 5.2 times compared with re-computation. The company said the setup virtually eliminated tail latency spikes caused by network congestion.
In a second test with AI technology partner Pickles, SK Hynix ran a full voice agent pipeline covering speech-to-text, LLM reasoning and text-to-speech. Intermediate outputs were streamed through shared CXL memory instead of over RDMA. Time-to-first-audio fell from 0.67 seconds to 0.46 seconds, 1.46 times faster. End-to-end latency fell from 8.08 seconds to 5.73 seconds, 1.41 times faster.
The third scenario was trillion-scale graph neural network (GNN) training, which usually means splitting large graphs across server nodes and waiting on frequent RDMA synchronization. Putting graph structure and feature gathering in a unified MARS pool sped up graph sampling 17.57 times compared with RDMA-based setups because the pool gives direct access at constant latency.

Credit: DIGITIMES
SK Hynix said it is co-developing a 16-tool software stack for MARS with partners from industry, government and academia. Partners include Los Alamos National Laboratory, Pacific Northwest National Laboratory, Stanford, UT Austin, the University of Illinois, Virginia Tech, UC San Diego, Carnegie Mellon, Georgia Tech and UC Berkeley/Merced. The tools cover memory allocators, inter-node locking, KV cache tiering, in-memory stores, and Graph RAG engines.
SK Hynix closed the session with High Bandwidth Flash (HBF), a memory tier that sits between HBM and enterprise SSDs.
HBF stacks 16 NAND dies with logic dies on a silicon interposer. It delivers about half the bandwidth of HBM with 10 to 14 times the capacity, and SK Hynix says it is 3.7 times more energy efficient per bit than traditional enterprise storage tiers. In demonstrations with Llama 3 70B and DeepSeek R1 671B, HBF handled context windows of up to 19 million tokens per user, according to the company.
SK Hynix said it has released the public OCP HBF Specification v0.7 with SanDisk and industry consortia. It plans to finalize Specification 1.0 next year and expects HBF Cube hardware sampling to begin around 2028.