Server Memory · GBA Technology · Published 2026-09-07 · Updated 2026-10-18

AI Server Memory 2026: Beyond HBM to MRDIMM, SOCAMM2 and CXL

Memory in AI servers is splitting from a single tier of DDR5 main memory into several layers: HBM inside the GPU package, RDIMM and MRDIMM on CPU memory channels, SOCAMM2 based on low-power LPDDR5X, and expansion memory attached over CXL. Each layer makes different trade-offs in capacity, bandwidth, latency and power. This article reviews the specifications, generational changes and vendor progress of each memory tier in 2026, and how they divide the work in AI systems.

SK hynix memory modules and HBM3 / HBM3E packages | AI Server Memory 2026 — SK hynix memory modules and HBM3 / HBM3E packages

Key Takeaways

Tiers: HBM supplies GPU compute bandwidth; RDIMM and MRDIMM provide CPU main-memory capacity and bandwidth; SOCAMM2 delivers dense main memory at low power; and CXL adds an expandable capacity tier.
RDIMM: DDR5-8000 RDIMMs have entered volume production, and 3DS RDIMMs reach 256 GB per module using 32Gb dies.
MRDIMM: The first generation runs at 8,800 MT/s; the second-generation standard targets 12,800 MT/s, and JEDEC has begun work on a third generation, which a roadmap AMD presented at MemCon 2023 puts at 17,600 MT/s.
SOCAMM2: SK hynix and Samsung are in volume production of 192 GB modules, and Micron has sampled 256 GB modules to customers.
HBM: HBM4 widens the interface to 2,048 bits with 36 GB in a 12-high stack; 12-high HBM4E samples reach 48 GB and 3.6 TB/s per stack.

Memory Tier Overview

Technology	Location	Unit capacity	Speed or bandwidth	2026 status
HBM3E	Inside the GPU package	8-high 24 GB / 12-high 36 GB (per stack)	About 1.2 TB/s per stack	Volume production
HBM4	Inside the GPU package	12-high 36 GB / 16-high 48 GB (per stack)	2,048-bit interface, from 2 TB/s per stack	Volume production
HBM4E	Inside the GPU package	12-high 48 GB (per stack)	Up to 3.6 TB/s per stack	Sampling
SOCAMM2	Next to the CPU, replaceable module	192 GB / 256 GB	LPDDR5X	Volume production 192 GB; 256 GB sampling
DDR5 RDIMM	CPU memory channel	16 GB to 128 GB	Up to DDR5-8000	Volume production
DDR5 3DS RDIMM	CPU memory channel	128 GB / 256 GB	DDR5-6400 to DDR5-8000	Volume production
MRDIMM Gen1	CPU memory channel	32 GB to 256 GB	8,800 MT/s	Shipping
MRDIMM Gen2	CPU memory channel	Up to 128 GB (Samsung)	12,800 MT/s	Demonstrated JEDEC standard being finalized
CXL Type 3 memory	Attached over PCIe / CXL	96 GB to 256 GB	PCIe 5.0 x8	Shipping CXL 2.0; CXL 3.x in development

Capacity, Bandwidth and Latency: Three Different Problems

Adding HBM does not remove the pressure on main memory; capacity, bandwidth and latency are three problems that need to be handled separately.

Model weights stay in GPU memory, but datasets, preprocessing buffers, host processes and partially offloaded KV cache still need main memory. When several GPUs share one data-loading pipeline, DDR5 must both hold the working set and keep data flowing.

Capacity: When the working set exceeds main memory, the system pages or swaps to disk, which shows up as sudden, large latency spikes.
Bandwidth: Even with enough capacity, high-concurrency reads and writes can be limited by per-channel bandwidth, which shows up as throughput that stops scaling.
Latency: Inference services are sensitive to tail latency, and access costs differ depending on whether memory sits on a local channel, a CXL device or a remote NUMA node.

64 GB, 96 GB, 128 GB and 256 GB modules are capacity options, not a performance ranking. How the same total capacity is spread evenly across channels often matters more for bandwidth than the capacity of a single module.

DDR5 RDIMM and 3DS RDIMM: From 24Gb to 32Gb Dies

DRAM die density determines the capacity ceiling of a single RDIMM and whether stacking technology is needed.

DDR5 RDIMM is the main form of main memory in enterprise servers, with speeds progressing from DDR5-4800, 5600 and 6400 to DDR5-8000. Micron has demonstrated DDR5-8000 RDIMMs and begun volume production; Intel Xeon 6+ runs 12 channels of DDR5-8000, and AMD EPYC 9006 and Intel Diamond Rapids also support DDR5-8000.

On capacity, 24Gb dies enable non-power-of-two capacities such as 48 GB and 96 GB, while 32Gb dies allow 128 GB RDIMMs to be built without stacking. Higher-capacity 3DS RDIMMs stack DRAM using through-silicon vias (TSV): SK hynix's 256 GB server RDIMM built on 32Gb dies has completed Intel data center certification, and Micron announced in May 2026 that it is sampling a 256 GB DDR5 3DS RDIMM rated up to 9,200 MT/s.

Configuration also depends on the number of modules per channel. Some platforms support two DIMMs per channel (2DPC) for more total capacity, but the supported speed is usually lower than with one DIMM per channel (1DPC). High-bandwidth configurations are built on one module per channel with every channel populated evenly.

MRDIMM: First to Third Generation

MRDIMM accesses two ranks at once through multiplexing, nearly doubling per-channel transfer rates over RDIMM while still using DDR5 dies.

MRDIMM (Multiplexed Rank DIMM) is a JEDEC-standardized module. MRCD and MDB chips on the module let two ranks operate at the same time with interleaved output, so the host interface runs at twice the rate of the DRAM itself.

First generation: Two ranks of DDR5-4400 interleaved for an 8,800 MT/s host interface, used with Intel Xeon 6900P. Micron's first-generation modules span 32 GB, 64 GB, 96 GB, 128 GB and 256 GB, with standard-height versions requiring no 3DS stacked dies; tall form-factor versions run about 24% cooler under the same airflow thanks to their larger surface area.
Second generation: Two ranks of DDR5-6400 interleaved for 12,800 MT/s, about 60% higher than DDR5-8000 RDIMM. JEDEC's JC-40 and JC-45 committees are finalizing the standard; Samsung has demonstrated second-generation modules at up to 128 GB. The AMD EPYC 9006 SP7 platform and Intel Diamond Rapids both list 12,800 MT/s as their top memory speed, and Intel has said Xeon 6900P will add second-generation MRDIMM support.
Third generation: JEDEC has begun developing the third-generation standard; the roadmap AMD presented at MemCon 2023 targets 17,600 MT/s.

MRDIMM increases how much data each channel can supply. The benefit appears mainly in bandwidth-bound, high-concurrency workloads; paging caused by insufficient capacity cannot be solved by raising transfer rates. Second-generation modules also top out below the largest 3DS RDIMMs, so capacity and bandwidth need to be weighed together.

CXL Memory Expansion: Adding Capacity Beyond the Channels

CXL Type 3 memory connects over the PCIe physical layer to add capacity without adding CPU memory channels, forming a larger memory tier with higher latency.

CXL memory expansion modules use the CXL.mem protocol, typically in EDSFF E3.S or add-in card form, connected over PCIe 5.0 x8. The operating system sees them as an additional memory node; because access passes through a CXL controller, latency is higher than local DIMMs, so they are usually paired with memory tiering that keeps hot data in local DIMMs and warm data in CXL memory.

Samsung: CMM-D is a CXL 2.0 module in 128 GB and 256 GB capacities with up to 36 GB/s of bandwidth. The CMM-B memory pooling appliance connects up to 24 E3.S CMM-D modules. Samsung also plans volume production of CMM-D 3.0 with CXL 3.2 within the year.
SK hynix: CMM-DDR5 is a CXL 2.0 module; the 96 GB version has completed customer validation, with a 128 GB version also offered, and samples of a second-generation CMM-DDR5 using CXL 3.2 have been shown publicly.
Micron: The CZ120 and CZ122 families of CXL 2.0 memory expansion modules come in 128 GB and 256 GB capacities.

CXL 3.x raises the rate to 64 GT/s (PCIe 6.0) and adds multi-level switching and memory sharing. On the platform side, Intel Xeon 6+ provides 64 lanes of CXL 2.0, Intel Diamond Rapids supports CXL 3.0 and NVIDIA Vera supports CXL 3.1.

SOCAMM2 and LPDDR5X: Low-Power Main Memory Enters the Server

SOCAMM2 turns LPDDR5X, originally used in mobile devices, into a replaceable server module that delivers dense main memory with lower power and a smaller footprint.

SOCAMM2 (Small Outline Compression Attached Memory Module 2) is secured to the motherboard with a compression connector, combining signal integrity with serviceability. The previous generation of AI host CPUs mostly soldered LPDDR5X directly to the board, fixing capacity at the factory; SOCAMM2 allows LPDDR5X main memory to be configured and replaced as needed.

SK hynix: Began volume production of 192 GB SOCAMM2 in April 2026 using 1c-process LPDDR5X. SK hynix says it offers more than twice the bandwidth of conventional RDIMM with over 75% better energy efficiency.
Samsung: In volume production of 192 GB SOCAMM2, using low-temperature solder technology to address module warpage. Samsung says it delivers up to 2.6 times the bandwidth of DDR-based server memory with more than 70% better energy efficiency.
Micron: Sampled 256 GB SOCAMM2 to customers in March 2026, built on the industry's first monolithic 32Gb LPDDR5X die, with power and size about one-third of a conventional RDIMM.

On a shipping platform, NVIDIA Vera supports up to eight SOCAMM2 modules per CPU, for up to 1.5 TB of capacity and up to 1.2 TB/s of bandwidth. SOCAMM2 requires a motherboard and CPU designed for its interface and cannot be installed in RDIMM slots; the two follow different platform paths.

HBM: From HBM3E to HBM4E

HBM stacks DRAM with through-silicon vias and sits in the same package as the GPU; HBM4 doubles the interface width, the largest architectural change in HBM to date.

HBM3E: 1,024-bit interface with about 1.2 TB/s per stack; 24 GB in an 8-high stack and 36 GB in a 12-high stack.
HBM4: JEDEC published the standard in April 2025, widening the interface to 2,048 bits and 32 channels; at the standard 8 Gbps, each stack delivers 2 TB/s. Vendor parts already exceed 11 Gbps per pin, with Micron samples delivering more than 2.8 TB/s per stack. A 12-high stack holds 36 GB, and SK hynix showed a 16-high 48 GB stack at CES 2026. Samsung began commercial shipments in February 2026, and SK hynix began mass shipments in the second quarter of 2026.
HBM4E: Samsung shipped the industry's first 12-high HBM4E samples in May 2026, with 48 GB per stack, 3.6 TB/s of bandwidth and pin speeds up to 16 Gbps; SK hynix sampled HBM4E to major customers in June 2026.

Starting with HBM4, the base die at the bottom of the stack moves to a foundry logic process: SK hynix's HBM4 uses a base die built on TSMC's 12nm logic process. The base die can integrate more logic functions, and HBM is moving from a standard part toward customization for each accelerator. HBM is part of the accelerator package and cannot be replaced or expanded by users; for the HBM configuration of each GPU, see Enterprise AI GPUs 2026.

Platform Memory Support

Platform	Memory type	Channels / modules	Top speed or bandwidth
AMD EPYC 9006 (SP7)	DDR5 RDIMM, MRDIMM	16 channels	DDR5-8000; MRDIMM 12,800 MT/s
AMD EPYC 9006 (SP8)	DDR5 RDIMM, MRDIMM	8 channels, up to 2 DIMMs per channel	DDR5-8000; MRDIMM 12,800 MT/s
Intel Xeon 6+	DDR5 RDIMM	12 channels	DDR5-8000
Intel Xeon 6900P	DDR5 RDIMM, MRDIMM	12 channels	DDR5-6400; MRDIMM 8,800 MT/s
Intel Diamond Rapids	DDR5 RDIMM, MRDIMM	16 channels	DDR5-8000; MRDIMM 12,800 MT/s
NVIDIA Vera	SOCAMM2 (LPDDR5X)	Up to 8 modules	1.2 TB/s per CPU
AmpereOne M	DDR5 RDIMM	12 channels, 1 DIMM per channel	DDR5-5600

For core counts, PCIe and overall architecture of each platform, see Server CPUs 2026.

Evaluation Framework

Fixing CPU, firmware and power settings before changing the memory configuration is how gains can be correctly attributed to memory itself.

Dimension	Metrics
Capacity headroom	Peak working set, paging and swap events
Sustained bandwidth	Sustained read/write bandwidth per channel and overall under real workloads
Tail latency	p99 latency of inference services, and access differences between local, CXL and remote NUMA memory
Power	Power per GB and per GB/s, and the system power ceiling
GPU stalls	Share of time GPUs spend waiting for data
Configuration	Channel population, DIMMs per channel, module type and firmware settings

If measurements show the limit lies in storage or networking, faster memory may not bring a matching improvement; confirming which layer causes GPU stalls comes before deciding how to adjust the memory configuration.

FAQ

How does MRDIMM differ from RDIMM?: Both use DDR5 dies. MRDIMM adds multiplexing chips so two ranks operate at once, making the host interface rate twice the DRAM rate: 8,800 MT/s in the first generation and 12,800 MT/s in the second. RDIMM currently tops out at DDR5-8000 but reaches higher capacities per module.
Can SOCAMM2 replace RDIMM?: They follow different platform paths. SOCAMM2 is based on LPDDR5X and requires a motherboard and CPU designed for its interface, such as NVIDIA Vera; it cannot be installed in RDIMM slots.
What are the main differences between HBM4 and HBM3E?: HBM4 widens the interface from 1,024 to 2,048 bits, raising per-stack bandwidth from about 1.2 TB/s to more than 2 TB/s; stacks can reach 16 high, and the base die moves to a logic process starting with HBM4.
What is CXL memory suited for?: It adds capacity without adding CPU memory channels, suiting data that needs large capacity but tolerates higher latency, such as large in-memory databases, data preprocessing and KV cache. Frequently used data stays in local DIMMs.

Sources

Last updated: 2026-10-18