The H200 looks like an H100 successor on the spec sheet. It isn’t, exactly.

Compute throughput is identical. Same 989 TFLOPS BF16. Same 1,979 TFLOPS FP8 dense. Same Hopper architecture. Same TDP. Same NVLink bandwidth.

The H200 thesis is memory — 76% more capacity (141 GB vs 80 GB) and 43% more bandwidth (4.8 TB/s vs 3.35 TB/s). Whether that 25–30% price premium pays back depends entirely on whether memory is your binding constraint.

This guide gives you the framework to decide. For the broader generation comparison including A100, see A100 vs H100 vs H200.

TL;DR

Upgrade to H200 if:

• Your inference workload uses long context (≥ 64K tokens) on large models (70B+)
• You’re memory-bandwidth-bound at H100 (KV cache reads dominate latency)
• You serve mixture-of-experts (MoE) models like DeepSeek V3 or Mixtral
• Single-node serving of very large models (200B+) where H100 forces multi-node parallelism

Stay on H100 if:

• You’re training (compute-bound, memory pressure managed via parallelism)
• You serve small/medium models or short contexts
• Your H100 utilization isn’t memory-bandwidth-saturated
• You’re acquiring fleet at scale and the $7K–$10K-per-GPU premium adds up

The honest median answer for most teams: H100 is still the right purchase in 2026 unless you have a specific memory bottleneck.

What’s actually different about H200

The full spec delta between H100 SXM5 and H200 SXM5:

This is an unusual delta in NVIDIA’s history. Generation-over-generation upgrades (P100→V100, V100→A100, A100→H100) typically deliver substantial compute increases. H100→H200 delivers zero compute increase. It’s a memory-only upgrade.

That’s not a flaw — NVIDIA correctly identified that for many AI workloads in 2024–2026, memory had become the binding constraint, not compute. H200 is a targeted product for those workloads. For other workloads, it’s overpriced H100.

Why memory bandwidth matters more in 2026

When LLM inference happens, the GPU does two things repeatedly: it reads model weights from memory, and it computes attention over the conversation context (the KV cache). Both are memory-bandwidth-intensive.

For a 70B model running in FP8:

• Weights: ~70 GB to read once per token (or once per batch with proper batching)
• KV cache: scales with context length and concurrent requests; can easily reach 30–50 GB at long contexts and high concurrency

At long contexts (64K+) with high concurrent batching, the GPU spends most of its time reading from memory, not computing. Compute throughput doesn’t matter when the GPU is bandwidth-starved.

H100 hits the bandwidth wall before its compute is fully utilized for this class of workload. H200, with 43% more bandwidth, lets the same compute deliver 30–40% more useful tokens-per-second.

For workloads that aren’t bandwidth-bound — training, short-context inference, small models — the extra bandwidth is unused. You paid for it; you don’t see it.

When H200 wins: workload-by-workload

Long-context inference (≥ 64K context, large models)

This is the H200’s strongest case. KV cache pressure at long context tilts the bottleneck firmly toward memory. H200 delivers materially better tokens/sec per GPU at the same batch size, and the gap widens as context length grows.

Worked example: serving Llama 3 70B at 128K context, batch 4 concurrent users.

H200 wins on cost per token even at 25% higher hourly rate, because the throughput improvement more than compensates.

Mixture-of-experts (MoE) models

MoE architectures (DeepSeek V3, Mixtral 8x22B, Qwen MoE) have many parameters but activate only a subset per token. The “many parameters” creates memory pressure; the “active subset” means compute is underused. H200’s larger memory and bandwidth fit this asymmetry exactly.

Single-node serving of very large models (200B+)

A 405B model in FP8 weights ~400 GB. On 8-GPU H100 nodes, that’s 50 GB per GPU just for weights — leaves 30 GB per GPU for KV cache, activations, and headroom. Tight. H200’s 141 GB per GPU gives 90 GB per GPU after weights, removing the constraint.

For models that won’t fit comfortably on a single 8×H100 node but do fit on 8×H200, the operational simplicity (single-node serving vs cross-node parallelism) is itself worth meaningful cost.

High-concurrency serving

Serving more concurrent users per GPU means amortizing the GPU cost across more requests. H200’s extra memory enables higher concurrent batch sizes for large models, which directly improves cost-per-request economics.

When H100 still wins: workload-by-workload

Training (any scale)

Training is compute-bound. Memory pressure during training is real but managed via parallelism strategies (tensor, pipeline, data, sequence) that already work fine on H100 80GB. The 989 TFLOPS compute is the binding constraint, and H200 doesn’t move it.

For frontier-model training, fleets are still being built primarily with H100 (and increasingly H200 mixed in for specific stages). Pure-H200 training fleets are uncommon because the compute parity makes the price premium hard to justify.

Short-context inference (≤ 8K context)

At short contexts, KV cache is small. Memory bandwidth doesn’t bind. H100 delivers the same throughput per dollar.

Small and medium models (≤ 13B)

These models fit comfortably in 80GB with room for substantial KV cache. H200’s extra memory is wasted. Compute parity means H100 wins on cost per token.

Cost-sensitive fleet builds

When buying 100+ GPUs for general-purpose workloads, the $7K–$10K per-GPU premium times the fleet size becomes a $700K–$1M decision. Unless your specific workload mix needs H200, H100 wins on capex.

The break-even calculation

The economic question: at what point does H200’s higher cost pay back via better throughput?

Assume you serve a long-context inference workload where H200 delivers 35% more throughput than H100 (a typical advantage at 64K+ context).

Per unit of useful throughput, H200 is 5% cheaper despite the higher sticker price. The premium pays back as long as the throughput delta materializes.

In cloud rentals, the math is similar:

H200 wins on effective cost — but only when throughput advantage materializes. For workloads where it doesn’t (training, short context, small models), the same calculation flips H100’s way.

For pure H200 pricing (without the comparison), see H200 Price: What It Actually Costs in 2026.

The upgrade question for existing H100 fleets

If you operate H100s already, the upgrade question is different from the new-buyer question. The trade-off is:

Cost of upgrade: new H200 capex (or rental switch) plus any operational disruption. Benefit: the throughput advantage on workloads that materialize it.

A common pattern in 2026: teams keep their existing H100 fleet for training and short-context inference, and add a smaller H200 fleet specifically for long-context inference workloads. This mixed-fleet approach gets H200’s benefits where they matter without paying the premium across the board.

If you’re considering owning H200 vs renting from cloud, see the dedicated analysis H200 vs Cloud Pricing: When Does Owning Make Sense?

What this means for the supply side

Mercatus tracks both H100 and H200 pricing in real time across 22+ providers. The cross-provider spread for H200 is similar to H100 (~2.8×) — meaning long-tail providers offer H200 at $2.50–$3.50/hr while hyperscalers price the same SKU at $4.50–$7.00/hr. That spread, combined with the H200’s specific workload advantages, makes provider choice as important as GPU choice.

For operators with H200 inventory (or considering acquiring some), the rapid growth of long-context inference workloads creates real demand. Long-context inference revenue per GPU-hour exceeds short-context inference by 30–50% in 2026 — H200 fleets serving that workload class earn back their premium quickly. Listing H200 capacity on Mercatus reaches the buyers specifically searching for memory-bandwidth-rich inference.

→ Become a Provider

For broader supply-side thesis, see The Open AI Compute Economy.

Frequently Asked Questions

Is H200 faster than H100?

For compute-bound workloads (training, dense computation): no, identical throughput. For memory-bandwidth-bound workloads (long-context inference, large models): yes, 30–40% faster on tokens-per-second. The answer depends entirely on workload.

How much more does H200 cost than H100?

OEM pricing: 25–30% premium ($32K–$40K vs $25K–$30K). Cloud pricing: similar 25% premium ($2.50–$4.50/hr vs $1.99–$3.50/hr). For specific workloads, the throughput advantage offsets — for others, it doesn’t.

Should I upgrade my existing H100 fleet to H200?

Most teams shouldn’t fully upgrade. The pattern that works in 2026 is mixed fleet: keep existing H100s for training and short-context, add H200s for long-context inference and serving very large models. Pure H200 fleets are rarely cost-justified.

Will Blackwell make H200 obsolete?

Eventually, yes — but Blackwell (B100, B200) supply is constrained through 2026 and pricing is high. For most teams, H200 has an 18–24 month sweet spot before Blackwell economics flip.

Where do I find H100 and H200 cloud pricing across providers?

Mercatus GPU Index tracks live cross-provider pricing for both. The 2.5–2.8× cross-provider spread is real — the same H100 or H200 SKU prices very differently at different providers.

Can I sell my idle H200 capacity?

Yes. If you operate H200 inventory and have unused hours, Mercatus lets you list inference capacity. Long-context inference and large-model serving are the highest-value workload categories on the buyer side.

→ Become a Provider

Methodology

Performance estimates derived from public benchmarking data (NVIDIA, ML Commons MLPerf, third-party throughput studies) and Mercatus aggregated pricing data. Cloud pricing reflects Mercatus GPU Index May 2026 snapshot. Throughput claims for long-context inference are workload-dependent — we recommend benchmarking your specific model and context length before acquiring at scale.

Last verified: 2026-05-04.