Ask what is compute and you get two answers. In everyday IT, compute is the raw processing power that runs software: the chips, memory, and machines that turn instructions into results. In the age of large AI models, the word has narrowed. Compute now means the specialized processing power used to train and run AI systems, counted in the number of math operations a chip can perform. It has quietly become the single resource that decides who can build frontier AI, and for most of 2024 through 2026 it has been in short supply. This guide explains the unit, the hardware, the market, and why the shortage matters.
What is compute, in plain terms
The base unit is the FLOP, short for floating-point operation: one arithmetic calculation on a decimal number. Modern AI is arithmetic at absurd scale. Training a large model means running septillions of these operations, so the field measures a training run in total FLOP and rates hardware in FLOP per second. A chip rated at one petaFLOP performs 10^15 operations every second.
Two figures set the scale. The research group Epoch AI estimates that GPT-4 was trained on roughly 2e25 FLOP, at an amortised hardware and energy cost near 40 million dollars (Epoch AI). And the training compute behind frontier models has grown about 4 to 5 times per year since 2010. Nothing else in computing has scaled that fast for that long, which is why "compute" went from a background utility to a boardroom word.
GPUs, accelerators, and why AI outgrew the CPU
A CPU, the central processing unit, is a generalist. It handles a few complex tasks quickly and runs the operating system, the database, and most business software. AI has a different shape. Training and inference are dominated by matrix multiplication, the same simple operation repeated across billions of numbers at once. That work suits a GPU, the graphics processing unit, which trades a handful of fast cores for thousands of simpler ones running in parallel.
The broader category is the AI accelerator: any chip specialized for this math. It covers NVIDIA and AMD GPUs, Google’s Tensor Processing Units, and the custom ASICs that hyperscalers design in house. NVIDIA’s current generation, Blackwell, packs 208 billion transistors on a TSMC 4NP process (NVIDIA). Raw math is only half the story: an accelerator also needs high-bandwidth memory (HBM) to feed the cores, and the fastest chips are increasingly limited by memory bandwidth rather than arithmetic. Progress here rides on ever-smaller transistors, the frontier we cover in IBM’s sub-1nm chip research. The same acceleration is arriving on laptops and handhelds through parts like the Ryzen AI Max+ 395, which put a neural engine next to the CPU.
Training versus inference: two different appetites
Compute gets spent in two distinct phases. Training is the one-time, brutally expensive process of building a model: feeding it data and adjusting billions of parameters until it works. Inference is what happens every time you use that model afterward: each prompt, each generated token, each answer.
The two have opposite profiles. Training is a huge burst on a tightly connected cluster for weeks or months. Inference is a steady, endless stream of smaller requests that has to stay cheap and fast at scale. Inference now accounts for roughly two-thirds of AI compute demand by several analyst estimates, because a model is trained once but queried billions of times. That is why the economics of serving a model come down to token math, which we unpack in input versus output tokens.
The split also shapes the hardware. Training rewards raw throughput and fast interconnect; inference rewards low latency, high memory bandwidth, and a low cost per token. That divergence is exactly why so many custom chips target inference specifically. It is the larger, steadier, more predictable share of the bill, and shaving a fraction of a cent off each response compounds across billions of requests.
Why compute is the gating resource for modern AI
For today’s leading labs, ideas are rarely the bottleneck; compute is. A lab’s ability to train a more capable model is capped by how many accelerators it can secure, how fast it can wire them together, and how much power it can draw. Epoch AI’s published projection is that the largest training runs will cost more than a billion dollars by 2027, on a growth rate of about 2.4 times a year. Nobody outside the labs knows what a 2026 frontier run actually costs, and figures in circulation are inference from that trend rather than disclosure.
That reframes AI as an infrastructure business as much as a software one. The scarce inputs are capital, chips, and electricity, and access to them increasingly decides who competes at the frontier. It also changes the buyer’s question from "which model is smartest" to "can I get enough capacity, at a price that works, for the workload I actually have." Treating compute as a budgeted, forecast resource is now part of the job, a discipline we discuss in managing AI investments in the agentic era.
Increasingly the ceiling is not chips but power. A large training cluster draws tens or hundreds of megawatts, and the newest buildouts are measured in gigawatts: AMD’s announced capacity agreements with Meta and OpenAI are framed in gigawatts of accelerators rather than in units. Electricity, grid connections, and cooling now sit on the critical path for AI, and siting a data center near cheap, reliable power has become a competitive advantage in its own right.
The compute market: NVIDIA, the custom-silicon wave, and the cloud
The supply side has three layers.
NVIDIA sits at the center. Analyst estimates put NVIDIA at roughly 75 to 80 percent of AI accelerator revenue in 2026, down from a peak near 87 percent in 2024. The scale is hard to overstate: NVIDIA reported 75.2 billion dollars in data-center revenue for the quarter ended April 2026, up 92 percent year over year. Its next architecture, Rubin, is slated to ship in the second half of 2026 with HBM4 memory, according to chief executive Jensen Huang. NVIDIA’s real moat is arguably software: the CUDA platform that most AI code is written against.
AMD is the credible second source. Its Instinct line, the MI350 and the MI400 series, launched in July 2026, with MI450 confirmed on a 2nm process, targets inference first and holds an estimated 5 to 7 percent of accelerator revenue, with reported supply deals involving Meta and OpenAI.
The custom-silicon wave is the real shift. The largest cloud providers now design their own accelerators to cut cost and reduce dependence on NVIDIA: Google’s TPU v7 (Ironwood), Amazon’s Trainium 3, Microsoft’s Maia 200, and Meta’s MTIA, which we cover in Meta’s MTIA AI chips. Model builders are joining them, as with the in-house accelerator we examine in OpenAI’s custom chip. One telling data point: in a joint announcement with Google Cloud in October 2025, Anthropic said it would gain access to up to one million TPU chips, with well over a gigawatt of capacity coming online during 2026. That is a contracted entitlement rather than a description of hardware running today. The catch is that these chips are captive. You cannot rent a Trainium or a TPU as bare silicon; you consume them only through their owner’s cloud. For everyone else, the practical universe of rentable AI compute is still NVIDIA first, AMD second, and a handful of specialist clouds reselling both.
Scarcity, cost, and the packaging bottleneck
The shortage of the last two years was rarely about making logic chips. The choke points sit downstream: advanced packaging (TSMC’s CoWoS, which stitches the GPU to its memory) and the supply of HBM itself. NVIDIA reportedly booked around 60 percent of TSMC’s 2026 CoWoS capacity, which constrains how many accelerators everyone else can ship, according to industry reporting.
Scarcity shows up directly in price. Renting a previous-generation H100 GPU ran roughly 2 to 3 dollars per GPU-hour across independent clouds in 2026, while a current Blackwell B200 sat nearer 5 to 6 dollars, per public pricing trackers. Prices ease as new supply lands and spike when a new model generation triggers a land grab. For any team running real workloads, that volatility makes compute a line item to forecast, not an afterthought.
Lead times compound the problem. Ordering the newest accelerators in volume can mean waiting months, so access often comes down to relationships and pre-commitments rather than a checkout page. The result is a market where the constraint is frequently allocation, not budget.
Cloud versus on-prem: how organizations actually buy compute
Almost no one outside the largest labs buys racks of accelerators outright. The realistic options are a spectrum.
- Public cloud, on-demand: rent by the hour or the token from AWS, Google Cloud, Azure, or a specialist GPU cloud. Maximum flexibility, highest per-hour cost, no capital outlay.
- Reserved or committed capacity: pre-book GPUs for months or years at a lower rate. Cheaper per hour, but you pay whether you use it or not.
- Spot or preemptible: use spare capacity at a steep discount, accepting that the provider can reclaim it. Good for fault-tolerant inference and interruptible training.
- On-prem or colocation: own the hardware for steady, heavy, latency- or data-sensitive workloads. Lower long-run cost at high utilization, but you carry the power, cooling, and depreciation.
For most organizations the deciding factor is not the silicon but the workload: how steady it is, how sensitive the data is, and how well the team can keep expensive hardware busy. Those are implementation questions, which is where our guide to AI implementation picks up.
Is compute really "the new oil"?
The phrase gets repeated because parts of it fit. Compute is a strategic input that a modern economy runs on, its supply is concentrated in a few hands, and it has become a matter of national policy, with export controls on advanced accelerators now a standard tool of statecraft. Capital is pouring into data centers and power the way it once flowed to refineries.
Governments have taken the analogy literally. Nations are funding domestic AI compute the way they once secured energy supply, and access to advanced accelerators is now negotiated between states. For a business the practical takeaway is narrower but real: the availability and price of compute can shift with policy, not only with technology, so a sourcing plan that assumes frictionless global supply is a fragile one.
The analogy also breaks in useful ways. Oil is consumed once and depleted; compute is durable, and every generation of chips does far more work per watt and per dollar than the last. Efficiency gains in models and hardware can blunt demand as fast as scarcity inflates it. A better way to hold the framing: compute is the strategic input of the AI era, but it is a moving target, not a fixed reserve. Understanding what it is, how it is measured, and who controls it is now basic literacy for anyone making technology decisions.
Frequently Asked Questions
What is compute in simple terms?
Compute is processing power: the chips and machines that carry out calculations. In an AI context it specifically means the specialized hardware, mostly GPUs and AI accelerators, used to train and run models, along with the amount of mathematical work that hardware can do.
What is a FLOP, and why does it matter for AI?
A FLOP is a single floating-point operation, one arithmetic calculation on a decimal number. AI training and inference are made of enormous numbers of these operations, so total FLOP measures the size of a training run and FLOP per second measures how fast a chip is. Epoch AI estimates GPT-4 took about 2e25 FLOP to train.
What is the difference between a CPU and a GPU for AI?
A CPU has a few powerful cores tuned for varied, sequential tasks. A GPU has thousands of simpler cores that run the same operation in parallel, which suits the matrix math behind neural networks. That parallelism is why GPUs and other accelerators, not CPUs, do the heavy lifting in AI.
What is the difference between training and inference?
Training builds a model, a one-time and very expensive process of adjusting billions of parameters. Inference runs the finished model to answer prompts. Training is a large burst of compute; inference is a steady stream that must stay cheap at scale, and it now accounts for the majority of AI compute demand.
Why is AI compute so scarce and expensive?
The limits are usually not in making the chips but in advanced packaging and high-bandwidth memory, both of which have constrained supply. Add surging demand from every major lab and cloud, and prices for renting accelerators stay high and volatile. Capacity, not model design, is often the binding constraint.
Does NVIDIA control the entire AI chip market?
No, but it dominates it. Analyst estimates put NVIDIA at roughly three-quarters of AI accelerator revenue in 2026, down from a 2024 peak. AMD is a growing second source, and the largest cloud providers now build custom chips such as Google’s TPUs and Amazon’s Trainium, though those are used only inside their own clouds.
Should my organization use cloud compute or buy its own?
Most organizations rent. Public cloud gives flexibility with no upfront cost; reserved capacity lowers the hourly rate for predictable workloads; owning hardware only pays off for steady, heavy use at high utilization. The right answer depends on how constant your workload is and how well you can keep costly hardware busy.
Do I need to think about compute if I only use AI through an API?
Indirectly, yes. Compute scarcity is what sets API prices, what determines whether a provider has capacity when you need it, and why rate limits exist at all. It also explains why cheaper, smaller models keep appearing: providers are trying to move routine work off the scarce hardware. You never provision anything, but you feel the constraint in pricing, availability and how quickly a new model reaches general access.
