An AI accelerator is a chip built to do one job faster and cheaper than a general-purpose processor can: the dense, low-precision matrix math that trains and serves machine learning models. As of August 2026 that market has three layers, and it has moved fast enough in the past year that most of the part numbers still circulating in trade coverage are a generation behind. NVIDIA has moved on from Blackwell. AMD has moved on from MI350. And the hyperscalers now ship enough silicon of their own to count as a third category rather than a footnote.
This is a map, not a recommendation. The goal is that you can read a vendor announcement or a cloud instance page and know what you are looking at.
What an AI Accelerator Actually Is
Strip away the branding and a data-center AI accelerator is two things bolted together. First, a large array of multiply-accumulate units tuned for low-precision number formats: FP8, FP4, and vendor variants like NVIDIA’s NVFP4 and AMD’s MXFP4. Second, a pool of high-bandwidth memory (HBM) stacked right next to the compute die.
The second part matters more than most coverage admits. Generating a token means reading model weights out of memory, and if the weights cannot move fast enough the compute units idle. That is why current spec sheets lead with memory capacity and bandwidth rather than peak FLOPS, and why HBM supply governs who ships what. We walked through how that translates into cost in our explainer on what compute actually means.
The category splits into two shapes. Merchant GPUs are sold to anyone: NVIDIA and AMD play here. Custom ASICs are designed by a company for its own fleet: Google, Amazon, Meta, Microsoft, and now OpenAI. A third shape, the dedicated inference processor, used to be a startup story and has just been absorbed into the first.
NVIDIA’s Current Line: Blackwell in Volume, Vera Rubin Ramping
Most deployed capacity today still runs on Blackwell and Blackwell Ultra, sold as the GB200 NVL72 and GB300 NVL72 rack systems. Hopper-era H100 and H200 parts remain in service and will for years. But NVIDIA’s current generation is Vera Rubin, announced at GTC on March 16, 2026 with seven new chips in full production: the Rubin GPU, the Vera CPU, an NVLink 6 switch, a ConnectX-9 SuperNIC, a BlueField-4 DPU, a Spectrum-6 Ethernet switch, and an LPU.
The flagship is Vera Rubin NVL72: 72 Rubin GPUs and 36 Vera CPUs in one NVLink domain. NVIDIA’s own claim is up to 10x higher inference throughput per watt than Blackwell at roughly a tenth of the cost per token. Treat that as a vendor figure until independent benchmarks land. On May 31, 2026 NVIDIA said Vera Rubin was ramping into full production, with production shipments starting in the fall and cloud availability through the second half of 2026. So for most of this year, buying NVIDIA still means buying Blackwell.
The seventh chip is the interesting one. NVIDIA Groq 3 LPU is an inference accelerator, deployed as an LPX rack of 256 LPU processors with 128 GB of on-chip SRAM and 640 TB/s of scale-up bandwidth, designed to sit alongside Rubin GPUs rather than replace them. It exists because of a non-exclusive inference technology licensing agreement announced by Groq on December 24, 2025, under which Groq’s founder and part of its team joined NVIDIA. Worth being precise here, because most coverage was not: it was a licensing and talent deal, not an acquisition. Groq remains an independent company and GroqCloud continues to operate.
The scale of NVIDIA’s position is easier to read from its own filings than from analyst estimates. For the quarter ended April 26, 2026, NVIDIA reported $75.2 billion in Data Center revenue, up 92% year over year.
Why CUDA Matters More Than the Silicon
Every serious analysis of this market arrives at the same conclusion: NVIDIA’s moat is software. CUDA has had twenty years of accumulation. The libraries built on it (cuDNN, NCCL, TensorRT, and the CUDA-X stack) are what frameworks target by default, what published research assumes, and what hiring managers screen for.
The switching cost is not a license fee. It is engineering time. A team that has written custom kernels, tuned a distributed training loop against NCCL, or built a serving path around TensorRT is not making a purchasing decision when it evaluates an alternative; it is scoping a porting project. That is why market share here is stickier than the underlying hardware advantage justifies. Tom’s Hardware put NVIDIA at roughly 70% of the AI chip market in a May 2026 analysis, though that is an analyst-style estimate rather than a reported figure, and no originating research firm is named.
AMD Instinct: The Main Merchant Alternative
AMD is the only other company selling a top-end AI accelerator to all comers. The volume part through most of 2026 has been the Instinct MI350 Series (MI350X and MI355X) on the fourth-generation CDNA architecture, with 288 GB of HBM3E per accelerator and 8 TB/s of memory bandwidth. AMD’s headline figure is up to 10.1 PFLOPS at MXFP4 and MXFP6 on the MI355X.
At its Advancing AI event on July 23, 2026, AMD launched the MI400 Series: the MI455X for large-scale AI and the MI430X for sovereign AI and HPC, both on HBM4. The system-level product is Helios, a rack pairing 72 MI455X accelerators across 18 four-GPU compute trays with sixth-generation EPYC "Venice" 9006 Series host CPUs, which AMD says is in production. AMD publishes rack-level numbers of 2.9 exaflops peak FP4, 31 TB of HBM4, and 1.7 PB/s of memory bandwidth, and claims 50% more HBM capacity than a Vera Rubin NVL72 rack. Those are AMD’s own comparisons against a competitor’s not-yet-shipping product, so weight them accordingly.
At the same event AMD announced a partnership with Cerebras that is more interesting than the MI400 launch itself, because it questions whether the rack is the right unit at all. The two companies will run Helios and the Cerebras Wafer-Scale Engine as a single disaggregated inference workflow, splitting the job by stage: Helios handles prompt processing and long context, where raw throughput matters, and the Wafer-Scale Engine handles token generation, where memory bandwidth and latency dominate. The claim is up to 5x higher tokens per second per watt, and the footnote deserves more attention than the headline. That figure comes from modelling by AMD and Cerebras rather than measurement, it is stated per kilowatt, it was derived on Kimi 2.6, and the baseline it beats is a Cerebras-only configuration rather than an NVIDIA one. Read properly, it says what AMD adds to Cerebras, not that the pair beats the market leader. The joint system is expected through Cerebras Cloud in the second half of 2026. The strategic signal is the real story: if the winning inference unit turns out to be two different silicon architectures in one rack, then a market map organised by vendor is already the wrong map.
The design wins are real and large: 6 gigawatts committed by OpenAI in October 2025, 6 gigawatts by Meta in February 2026, and up to 2 gigawatts of MI450-class capacity by Anthropic announced on July 22, 2026. Revenue tells a more sober story. AMD reported $5.8 billion in Data Center revenue for Q1 2026, up 57%, and that segment includes server CPUs. The accelerator-only gap with NVIDIA is wider than the 13x ratio between those two headline numbers suggests.
On software, ROCm has reached version 7.x with documented paths for PyTorch, JAX, vLLM, and SGLang. For teams living at framework level, porting is now genuinely routine. For teams with hand-written kernels, it is still a project.
The Custom-Silicon Wave
The third layer is AI accelerator silicon you cannot buy, only rent.
Google’s current generation is TPU7x, marketed as Ironwood, generally available with 192 GB of HBM per chip and pods of 9,216 chips. Its successors, TPU 8t for pre-training and TPU 8i for serving, were announced on April 22, 2026 and are not yet generally available. Amazon reached general availability on Trainium3 on December 2, 2025, with Trn3 UltraServers scaling to 144 chips at 144 GB of HBM3E each. Microsoft announced Maia 200 on January 26, 2026: 216 GB of HBM3E, 7 TB/s, clusters up to 6,144 accelerators, and explicitly framed as an inference part.
Meta said in March 2026 that MTIA 300 is in production and that it has deployed hundreds of thousands of MTIA chips for inference, with MTIA 400, 450, and 500 planned into 2027 and a co-development partnership with Broadcom announced in April 2026. We covered that program in Meta’s MTIA effort. OpenAI and Broadcom announced a collaboration on October 13, 2025 covering 10 gigawatts of OpenAI-designed accelerators with rack deployments starting in the second half of 2026, which we tracked in our piece on OpenAI’s own chip.
The aggregate trend is measurable. TrendForce projects ASIC-based AI server shipments at 27.8% of the market in 2026, and says ASIC shipment growth will outpace GPU growth without publishing the two rates. That is not NVIDIA losing; it is the total pie growing while the custom slice grows faster.
Training Silicon and Inference Silicon Are Diverging
The most useful structural observation of 2026 is that one chip is no longer expected to do both jobs well.
Training rewards raw throughput, huge coherent memory domains, and interconnect bandwidth. Inference rewards low latency, memory capacity for KV cache, and predictable cost per token at a given response time. Those pull in different directions, and every major vendor has now split the roadmap to acknowledge it. NVIDIA pairs HBM-rich Rubin GPUs with SRAM-rich LPUs. Google separated TPU 8t from TPU 8i. Microsoft built Maia 200 for inference. Meta’s deployed base is inference-heavy, though MTIA 300 is a training part, used for ranking and recommendations training and already in production.
For buyers this is the practical takeaway: benchmark the workload you actually run. An AI accelerator that wins on training throughput can lose badly on interactive serving, and the reverse is equally true. Our look at what local inference actually needs makes the same point at a much smaller scale.
The Real Constraint Is Packaging, Memory, and Power
Nobody is short of AI accelerator designs. They are short of three other things.
Advanced packaging is first. TSMC does not publish CoWoS wafer-per-month capacity, so every figure in circulation is an analyst estimate rather than a company number, and Tom’s Hardware reported in May 2026 that allocation runs to roughly 60% NVIDIA, 15% Broadcom, and 11% AMD. Packaging, not wafer fabrication, is now the binding constraint. Process nodes still matter, as we noted when IBM went sub-1nm, but they are no longer the bottleneck.
Memory is second. HBM4 is the current leading edge. TrendForce reported in January 2026 that SK hynix is expected to supply close to two-thirds of NVIDIA’s HBM4 demand, with Samsung reportedly targeting early HBM4 delivery, a date that appears to have slipped toward March. Micron hinted as early as August 2025 that its 2026 output would sell out. Treat blanket "everything is sold out" claims with suspicion; the verified position is tight allocation, not a uniform sellout.
Power is third, and increasingly decisive. Performance per watt and tokens per megawatt are the metrics vendors now lead with, because a data center’s power envelope is fixed long before its chip order is.
How to Choose an AI Accelerator
For almost every organization reading this, the honest answer is that you will rent, not buy. That reframes the question from "which chip" to "which instance, at what commitment, for which workload." A short checklist:
- Training or serving? They have different winners. Do not let a training benchmark sell you an inference platform.
- How deep is your CUDA dependency? If your stack is PyTorch, vLLM, and standard operators, you are portable. If it is hand-tuned kernels, you are not, and that has a dollar value.
- Does the model fit? Memory capacity per accelerator decides how many devices a model needs, which decides interconnect cost. Check this before checking FLOPS.
- What is the cost per million tokens at your latency target? Peak FLOPS is a marketing number. Cost per token at an acceptable time-to-first-token is the operating number.
- Can you actually get capacity? Allocation and lead time beat list price in a constrained market. Ask about contract length and what happens at renewal.
- Keep an exit. Staying at framework level is cheap portability insurance, and it is the same discipline we argued for in managing AI investments in the agentic era.
August 2026 leaves one dominant vendor with a software moat, one credible merchant challenger with real design wins and a narrower software story, and a growing block of custom silicon reached through a cloud console rather than a purchase order. None of that reverses in twelve months. What will change is which part number sits behind the instance you rent, which is a good argument for not building anything that cares.
Frequently Asked Questions
What is an AI accelerator?
An AI accelerator is a processor designed specifically for the matrix math behind machine learning, rather than for general-purpose computing. In practice that means a large array of low-precision multiply-accumulate units paired with high-bandwidth memory. GPUs, custom ASICs like Google’s TPU, and inference processors such as LPUs all fall into the category.
Is an AI accelerator the same thing as a GPU?
Not quite. All current data-center GPUs are AI accelerators, but not all AI accelerators are GPUs. Google’s TPUs, AWS Trainium, Meta’s MTIA, and Microsoft’s Maia are application-specific chips with no graphics heritage. The GPU label survives mostly for historical reasons.
What is NVIDIA’s current data-center accelerator?
Vera Rubin, announced at GTC on March 16, 2026 and built around the Rubin GPU and Vera CPU. NVIDIA said on May 31, 2026 that the platform was ramping into full production with shipments starting in the fall of 2026. Blackwell and Blackwell Ultra, sold as GB200 NVL72 and GB300 NVL72, remain the volume products in the meantime.
What is AMD shipping right now?
The Instinct MI350 Series (288 GB of HBM3E, 8 TB/s) has been the volume part, and AMD launched the MI400 Series, MI455X and MI430X, on July 23, 2026 alongside its Helios rack system. AMD reported $5.8 billion in Data Center revenue for Q1 2026, a segment that also includes server CPUs.
Can AMD Instinct run software written for CUDA?
Not directly. AMD’s ROCm stack supports PyTorch, JAX, vLLM, and SGLang, so code written at framework level generally moves with modest effort. Code that calls CUDA-specific kernels or libraries has to be ported. That porting cost, not the hardware, is usually what decides whether a team switches.
Why do Google, Amazon, Meta, and Microsoft build their own chips?
Three reasons: cost per token at their volume, supply independence from a single vendor, and the ability to tune silicon to workloads they already know precisely. Meta has said its program is inference-first because it runs enormous, well-characterised ranking and recommendation inference. These chips are rarely sold externally, so they reach you as cloud instances.
What is HBM and why does it keep coming up?
High-bandwidth memory is DRAM stacked vertically and placed next to the compute die, connected by a very wide interface. It is what lets an accelerator stream model weights fast enough to keep its compute units busy. HBM4 is the current leading edge, and because only three suppliers make it at scale, its availability shapes the whole market.
Is NVIDIA’s lead narrowing?
Slowly, and mostly at the edges. TrendForce projects ASIC-based AI servers at 27.8% of 2026 shipments against 69.7% for GPUs, and says ASIC growth will outpace GPU growth, which suggests hyperscaler silicon is absorbing the marginal workload. But NVIDIA’s $75.2 billion Data Center quarter, roughly thirteen times AMD’s comparable segment, says the core has not moved.
