There are roughly two kinds of GPU clusters in 2026: ones running at 40% utilization (most of them) and ones running at 80%+ utilization (a few). The cost-per-useful-output gap between the two is roughly 2× — for the same hardware, same power, same colocation, same operations.
Utilization is the single largest controllable variable in AI infrastructure economics. It’s also the metric most teams measure wrong, plan around badly, and underinvest in optimizing. This article covers what utilization actually means, what number you should target, how it translates to cost, and why the new lever — monetizing idle capacity — changes the optimization calculus fundamentally.
TL;DR
- GPU utilization is the percentage of paid-for GPU-hours that produce useful output. Most teams measure something else (and overstate the number).
- Realistic 2026 cluster utilization sits at 40–70% for most institutional deployments. The gap to optimal is enormous.
- Every 10% utilization improvement on a 100-H100 cluster saves ~$200K/year in effective compute cost.
- The new lever in 2026: sell idle capacity as inference tokens through Mercatus. Idle GPU-hours that used to be sunk cost become revenue offsetting your fleet operating cost.
What “GPU utilization” actually means (and why most teams measure it wrong)
When teams say “our GPU utilization is 80%,” they usually mean one of three things:
Definition 1: nvidia-smi reported utilization
The number nvidia-smi shows. This is the percentage of the time interval during which at least one compute kernel was active on the GPU. It does not measure how much of the GPU was being used — only whether something was running.
A workload that uses 5% of the GPU’s compute capability and runs continuously will show 100% nvidia-smi utilization. This metric is misleading.
Definition 2: Streaming Multiprocessor (SM) occupancy
The percentage of the GPU’s parallel compute units actively executing work. This is the metric that most closely tracks “real” utilization for compute-bound workloads. SM occupancy is typically 30–50% of nvidia-smi-reported utilization. A cluster reporting “80% utilization” likely has 30–40% SM occupancy.
For training workloads, SM occupancy of 60%+ is good. For inference workloads, the number depends heavily on batch size and model architecture.
Definition 3: Memory bandwidth utilization
For memory-bound workloads (long-context inference, large model serving), the binding constraint isn’t compute — it’s how fast weights and KV cache can be read from memory. Memory bandwidth utilization is the relevant metric here. H200’s 4.8 TB/s vs H100’s 3.35 TB/s is meaningful precisely because memory is what binds.
Definition 4: Economic utilization
The most useful definition for cost analysis: the percentage of paid-for GPU-hours that produce business value. This factors in:
- Idle time (GPUs powered but not running compute)
- Waste compute (GPUs running unproductive workloads — failed jobs, debugging, abandoned experiments)
- Underutilized compute (running but at low SM occupancy)
Most institutional deployments have economic utilization in the 30–55% range, even when their nvidia-smi numbers look acceptable.
Typical utilization rates by workload type
What good utilization looks like, by workload class:
| Workload | Typical economic utilization | What “good” looks like |
|---|---|---|
| Foundation model training (frontier-scale) | 50–65% | 70%+ requires aggressive scheduling |
| Fine-tuning runs | 60–80% | Easier to saturate single jobs |
| Production inference (well-tuned) | 60–75% | Continuous batching helps |
| Production inference (typical) | 30–50% | Variable demand kills utilization |
| Research and dev | 20–40% | Inherently sporadic |
| Mixed workload (most institutional) | 40–60% | Different jobs cannibalize each other |
The honest median for most 100-GPU institutional clusters: 50–60% economic utilization. Top quartile: 70–80%. Bottom quartile: 30–40%.
For analytics, see Mercatus Token Index data on inference throughput across providers — high-utilization providers are visible in the data.
How utilization translates to cost
For a 100-H100 cluster with $2.0M/year operating cost (per the own-side TCO pillar):
| Economic utilization | Effective $/GPU-useful-hour |
|---|---|
| 30% | $7.62 |
| 50% | $4.57 |
| 70% | $3.27 |
| 90% | $2.54 |
Moving from 50% to 70% utilization on a 100-GPU cluster saves roughly $1.30/useful-hour × 100 GPUs × 8,760 hours = ~$1.14M/year. Or stated as a rule of thumb: every 10 percentage points of utilization improvement saves about $200K/year on a 100-GPU H100 cluster.
This is the single largest cost lever in fleet economics. Larger than power optimization. Larger than colocation rate negotiation. Larger than OEM volume discounts.
For the cost-vs-utilization sensitivity table at full detail: 100 H100 Cluster TCO.
How to actually improve utilization
Five interventions, ranked by impact:
1. Continuous batching for inference (highest impact for serving)
Static batching (group requests into fixed-size batches) leaves GPUs underutilized between batches. Continuous batching (vLLM, TGI, recent versions of major inference servers) keeps the GPU productive across request boundaries, often pushing utilization 30–50% higher for the same hardware.
This is the single biggest improvement for inference-heavy workloads, and it’s mostly software-side.
2. Workload mixing and scheduling
Most clusters have multiple workload classes (training jobs, fine-tunes, inference traffic, batch processing). Smart scheduling places compatible workloads on the same GPUs at the same time, fills gaps in interactive workload with batch processing, and prevents resource fragmentation.
Tools: Run:AI, Nephele, Volcano, custom scheduling on Kubernetes. Effort: moderate. Impact: 10–20% utilization improvement typical.
3. Pre-emptive scaling for predictable patterns
Many workloads have predictable demand cycles (business hours, geographic sun-following, weekly patterns). Pre-emptively scaling capacity up before demand and down after avoids the lag of reactive autoscaling. Reactive autoscaling typically responds 10–20 minutes too late, leaving demand spikes underserved and capacity idle on the down-ramp.
4. Job admission control and queue management
Better job admission (deferring non-urgent jobs, preempting low-priority work) keeps GPUs fed. Many clusters waste utilization on poorly-prioritized work.
5. Hardware right-sizing
Running large models on small GPUs (or vice versa) hurts utilization. Right-sizing means the GPU’s compute and memory match the workload’s needs. Often this means using smaller GPUs more efficiently rather than upgrading.
For specific workload-to-GPU recommendations: A100 vs H100 vs H200.
The new lever: monetizing idle capacity
Through 2025, idle GPU capacity was unrecoverable cost. You paid for the GPU; if you didn’t use it, the spend was sunk. The optimization horizon ended at “improve internal utilization.”
In 2026, that’s no longer true. Idle capacity has commercial value. Mercatus’s open Spot Market lets any qualified operator list inference capacity for sale. Buyers route through a unified API; providers serve traffic from spare capacity that would otherwise sit idle.
For a 100-GPU H100 cluster running at 70% primary utilization:
// text
Primary workload utilization: 70% × 8,760 hr × 100 GPUs = 613K GPU-hr/yr
Idle capacity available: 30% × 8,760 hr × 100 GPUs = 263K GPU-hr/yr
Revenue from idle (at $2.00/hr): $526K/year
Effective cost reduction: ~26% of $2.0M operating costThe cluster’s effective utilization becomes 100% economically even though primary workload utilization is still 70%. Two revenue streams: the primary workload value and the offset revenue from idle capacity sale.
This changes utilization optimization from “improve internal utilization to reduce cost” to “improve internal utilization for primary value, and monetize the rest.” The new optimal is whichever produces more business value, not whichever fills more GPU-hours with primary work.
For details on becoming a Provider: Become a Provider. For the broader thesis on why this is the new structural lever: The Open AI Compute Economy.
What this means for the buy-vs-rent decision
Utilization is the central variable in the buy-vs-rent decision. The framework:
- High utilization (≥75%) + capacity monetization → owning wins decisively
- Mid utilization (50–75%) without monetization → reserved cloud capacity is competitive
- Low utilization (<50%) → cloud always wins; pay only for what you use
The capacity monetization lever shifts the threshold ~15 percentage points lower. Without monetization, you need 75%+ utilization to justify owning. With it, 60% is enough.
For the full framework: Buy vs Rent GPUs.
How utilization shows up in token prices
For API buyers (you don’t operate clusters), utilization upstream determines what you pay per token. Providers running at 70%+ utilization can offer per-token prices 20–30% below providers running at 40%. The cross-provider per-token spread you see on Mercatus Token Index reflects exactly this.
When you choose a provider for inference, you’re implicitly choosing their utilization story. High-throughput providers with continuous batching and good scheduling pass their efficiency through as lower prices. Low-utilization providers don’t.
For unified routing across providers (so you automatically benefit from the high-utilization ones): Mercatus Spot Market.
Frequently Asked Questions
What’s the average GPU utilization rate in 2026?
Real-world utilization across institutional AI deployments averages 40–70%. Top quartile reaches 70–80%; bottom quartile sits at 30–40%. The honest median for most clusters is 50–60% economic utilization — meaning roughly half of paid-for GPU-hours produce useful output.
How does GPU utilization affect cost?
Linearly and significantly. Effective $/GPU-useful-hour roughly doubles when utilization drops from 80% to 40%. Every 10 percentage points of utilization improvement saves approximately $200K/year on a 100-H100 cluster.
Is nvidia-smi the right metric for utilization?
No. nvidia-smi reports whether any kernel is running, not how much of the GPU is being used. SM occupancy and memory bandwidth utilization are better proxies for actual work. Economic utilization (paid-hours that produce business value) is the right metric for cost analysis.
How can I improve cluster GPU utilization?
Five high-impact interventions: continuous batching for inference, workload mixing and scheduling, pre-emptive scaling, better job admission control, and hardware right-sizing. Continuous batching alone often delivers 30–50% utilization improvement on serving workloads.
Can I monetize unused GPU capacity?
Yes. Mercatus Spot Market lets operators list inference capacity for sale. Idle capacity that would otherwise be sunk cost becomes revenue. For a 100-H100 cluster at 70% primary utilization, idle-capacity revenue at $2/hour can offset 25%+ of operating cost. → Become a Provider.
What utilization rate do top AI labs run at?
Frontier-model training runs typically achieve 50–65% during active training (the constraint is communication overhead, not GPU compute itself). Inference services at major API providers run higher — 60–80% on well-tuned serving stacks.
How does utilization relate to per-token pricing?
Providers with high cluster utilization can offer lower per-token prices because their fixed costs amortize over more useful output. Cross-provider price differences for the same model often reflect their underlying utilization economics. See Token Index for transparent cross-provider pricing.
Methodology
Utilization rate ranges and cost calculations derived from Mercatus aggregated provider data and institutional customer benchmarks. Cost-per-utilization sensitivity model assumes the 100 H100 Cluster TCO base case. Last verified: 2026-05-04.