AI GPUs 2026: H200 to Vera Rubin — Why More Isn't Always Faster

Competition among enterprise AI GPUs is expanding from single-card peak compute to HBM capacity, interconnect reach, data paths and rack-level design. What enterprises measure has shifted from "how fast is one card" to "how much work can a system complete under defined quality, latency and power conditions." This article reviews NVIDIA's progression from Hopper and Blackwell to Vera Rubin, AMD's progression from MI300 to the MI400 series, and the next-generation roadmaps of both.

Platform Overview

From Compute to the Data Path

The same GPU stall can have entirely different causes, and adding GPUs does not necessarily shorten service time.

An inference service that already fits its model in GPU memory can slow down as concurrency rises: the KV cache grows with the number of requests, and memory access and cross-GPU communication begin to compete for resources. A training job, meanwhile, may pause while writing checkpoints to storage. The first is bound by memory and interconnect; the second by the storage path.

This is why accelerator competition now extends to HBM, interconnect and rack design. Single-card peak figures describe the ceiling of one computation; what enterprises need to measure is the work a system completes under defined quality, latency and power conditions.

Hopper to Blackwell: Memory and Precision Change Together

From Hopper to Blackwell, the change is not only in compute speed but in whether a model fits in local GPU memory and which low-precision execution paths are available.

H100 and H200 share the Hopper architecture. H200 raises memory to 141 GB HBM3e, which directly affects whether a model fits entirely in a single GPU's memory and how many GPUs it must be split across. H20 is also a Hopper part, but its memory and compute configuration differ from H100 and H200, so comparisons are made at the level of the complete node, GPU count, drivers and software stack.

B200 and B300 (Blackwell Ultra) extend evaluation to low-precision paths such as FP8 and FP4, with B300 carrying 288 GB HBM3e per GPU. The throughput gains from lower precision depend on model quantization calibration, operator support and output quality; peak figures at different precisions reflect different conditions and should be compared at the same quality threshold.

GB200, GB300 and Vera Rubin: The Rack Becomes the Unit of Comparison

GB200 NVL72, GB300 NVL72 and Vera Rubin NVL72 shift the focus of evaluation from the single card to coordination within the rack.

GB200 NVL72 and GB300 NVL72 connect 72 Blackwell-family GPUs and 36 Grace CPUs over NVLink within a single rack. Vera Rubin NVL72 continues the rack-scale design with 72 Rubin GPUs and 36 Vera CPUs; sixth-generation NVLink in NVL72 provides up to 3 TB/s of interconnect bandwidth per GPU and 216 TB/s across the rack, with 288 GB of HBM4 per GPU as the new generation of GPU memory.

In rack-scale systems, the key metric is how many interconnect boundaries a job must cross, not just the total GPU count. A larger high-speed interconnect domain requires software scheduling, networking and data supply to keep pace before it translates into real throughput.

Vera Rubin Platform Components: Placing Each Part on the Data Path

Vera Rubin is a platform built from seven types of chips rather than a single GPU, and each component handles a different segment of the data path.

• Rubin GPU and Vera CPU: Compute and host coordination. Vera uses 88 custom Olympus cores and shares coherent memory with the GPU over NVLink-C2C.
• NVLink 6 Switch: The in-rack scale-up interconnect, allowing 72 GPUs to work together in an all-to-all topology.
• ConnectX-9 SuperNIC: Scale-out networking beyond the node, providing 1.6 Tb/s of network bandwidth per GPU.
• BlueField-4 DPU: Combines a 64-core Grace CPU with ConnectX-9 to handle networking, storage, security and infrastructure data movement.
• Vera BlueField-4 STX: Targets AI-native storage and context memory, supporting access to and reuse of long-context data such as the KV cache.
• Spectrum-6 Ethernet switch: Multi-rack Spectrum-X Ethernet networking with co-packaged optics.
• Groq 3 LPU: An inference accelerator for low-latency token generation.

In terms of system form, Vera Rubin NVL72 is a rack-scale system; HGX Rubin NVL8 is an eight-GPU platform that can be paired with Vera CPUs or x86 baseboards; and DGX Rubin NVL8 is a liquid-cooled eight-GPU system.

Groq 3 LPX is the inference acceleration rack of the Vera Rubin platform, with 256 LPUs, 128 GB of SRAM, 40 PB/s of memory bandwidth and 640 TB/s of scale-up bandwidth per rack. In the division of work, Vera Rubin NVL72 handles large-scale context processing while LPX handles latency-sensitive token decoding. For long-context, low-latency agentic inference, the evaluation metrics are end-to-end token latency, throughput, data movement and power.

AMD Instinct: From MI300 to MI400

AMD's path began with a lead in memory capacity and has moved toward low-precision compute, rack-scale reference designs and HPC-specific parts, with the ROCm software version a common condition in every generation's evaluation.

MI300X and MI325X (CDNA 3): Progressed from 192 GB HBM3 to 256 GB HBM3E, using large memory capacity to reduce model sharding.

MI350X and MI355X (CDNA 4): Carry 288 GB HBM3E and add low-precision compute capabilities. They can be placed in the same test design as H100, H200, B200 or B300; model porting, framework versions, operator coverage, collective communication and fault recovery together determine effective utilization.

MI455X and the Helios rack-scale reference design (CDNA 5): MI455X carries 432 GB HBM4 and targets large-scale inference and frontier model training. Helios is AMD's rack-scale reference design, built from 72 MI455X GPUs, 6th Gen EPYC CPUs and Pensando networking, with about 31 TB of HBM4 in total and up to 1.4 exaFLOPS at FP8 and 2.9 exaFLOPS at FP4. The unit of comparison is again the full coordination domain: host CPU, networking, shared memory domain, cooling and ROCm version are all test conditions.

MI430X (CDNA 5): Positioned for sovereign AI and HPC, with HBM4 and up to 288 TFLOPS of hardware FP64 performance, suited to environments that need both high-precision scientific computing and AI. MI430X is expected to be available in 2027 and has been selected for supercomputers including Discovery at Oak Ridge National Laboratory in the US, and Alice Recoque, Herder and LUMI-AI in Europe.

Next-Generation Roadmap

NVIDIA and AMD have both published accelerator roadmaps through 2028. These roadmaps can be used to assess future data-center requirements for power, liquid cooling, optical interconnect and software evolution in advance.

1. 2027Roadmap NVIDIA Rubin Ultra, using HBM4e and the Kyber rack architecture, covering NVL144 and NVL576 configurations.
2. 2027Roadmap AMD Instinct MI500 (CDNA 6, HBM4e) and the Helios 500 rack, paired with EPYC Verano.
3. 2027Launched AMD MI430X availability.
4. 2028Roadmap NVIDIA Feynman generation: a new GPU, Rosa CPU, LP40 LPU, BlueField-5 and CX10 SuperNIC, with Kyber-CPO introducing co-packaged optics for scale-up interconnect.
5. 2028Roadmap AMD Instinct MI600 and the Helios 600 rack, paired with EPYC Ferrara.

Power Density and Facility Requirements

As more accelerators are concentrated in the same coordination domain, power delivery and cooling become part of system design and need to be planned alongside platform selection.

The usable compute of a rack-scale solution depends on whether the facility can continuously remove heat and how much service capacity remains during maintenance. From GB300 NVL72 and Vera Rubin NVL72 to the Kyber architecture of Rubin Ultra, rack power density and liquid-cooling requirements rise with each generation. For systems already in operation, identifying whether GPU stalls originate in memory, networking or data supply is the basis for deciding which layer to optimize next.

Platform Evaluation Framework

Further Reading

• Server CPUs 2026: What Matters After 256 Cores? EPYC, Xeon, Vera
• Pro GPU Upgrades 2026: RTX PRO 6000 Blackwell vs RTX 6000 Ada

Sources

• NVIDIA Vera Rubin Platform
• NVIDIA Vera Rubin Platform Announcement
• NVIDIA NVLink and NVLink Switch
• NVIDIA Groq 3 LPX in Full Production
• NVIDIA Data Center Products
• AMD Instinct MI400 Series Launch
• AMD Instinct GPUs and Helios

Last updated: 2026-10-18