How Much Compute Do You Actually Need?

Every AI team eventually asks the same question, usually right before a deadline: “How many GPUs do we need?”

GPU count is not a fixed property of a model. It is the result of a set of workload constraints. Two teams can deploy the same 70B-parameter model and require different systems because one is fine-tuning against a fixed completion date while the other is serving a variable request stream under p95 latency and throughput targets. The right configuration is the smallest system that satisfies memory fit, time-to-completion, service-level objectives, and reliability requirements at an acceptable cost per completed workload.

Parameter count alone is a poor predictor of GPU requirements. It approximates weight storage, but the runtime footprint also includes gradients, optimizer states, saved activations, attention intermediates, temporary workspaces, and, for autoregressive inference, the KV cache. Architecture, storage and compute dtypes, microbatch size, sequence length, activation recomputation, concurrency, and the choice of data, tensor, pipeline, context, or expert parallelism all change per-rank memory, communication volume, kernel shape, and GPU utilization. NVIDIA’s Megatron Bridge performance guidance treats model architecture, hyperparameters, GPU count, GPU type, and parallelism topology as interacting variables that must be tuned together. There is no reliable one-variable formula.

This guide uses a workload-specific process: define the performance objective, build a per-GPU memory budget, benchmark the smallest valid configuration, measure strong-scaling efficiency and communication overhead, then separate steady-state demand from temporary burst demand. 

The six questions that determine your GPU count

Before comparing GPU specifications, define these six inputs. Each changes either the feasibility floor, the performance target, or the efficiency of the final system.

1. Start with the workload, not the hardware

Training, fine-tuning, batch inference, and online inference are different optimization problems. MLCommons makes the same distinction: MLPerf Training measures time to reach a defined quality target, while MLPerf Inference: Datacenter evaluates throughput under workload-specific latency constraints and request scenarios. The objective function changes, so the infrastructure decision changes with it.

For each workload, define the constraint you are solving and the metric that determines whether the configuration passes:

2. Will it fit? Build the memory budget first

GPU count has a hard feasibility floor: every rank or pipeline stage must fit within the HBM available to it after accounting for sharding, replication, and runtime headroom. Aggregate HBM is not a single pooled address space. A job can have enough total memory across a cluster and still fail because one rank, stage, or expert exceeds local device memory.

For training, budget memory separately for model weights, gradients, optimizer states and master weights, saved activations, attention tensors, communication buffers, kernel workspaces, the CUDA context, allocator fragmentation, and framework overhead. The per-GPU result depends on the distributed strategy: DDP replicates model state, FSDP and ZeRO shard selected states, tensor and pipeline parallelism partition the model, and context parallelism partitions sequence-dependent activations  The Hugging Face GPU memory anatomy shows why weight memory alone substantially understates training requirements.

For dense-model inference, begin with the storage required for the checkpoint weights:

Then estimate the KV cache for the decoder architecture and expected tokens in flight. For a conventional transformer cache, a useful first-pass approximation is:

The factor of 2 represents keys and values. Grouped-query and multi-query attention reduce KV-head count relative to standard multi-head attention, while tensor parallelism may shard cache state across ranks. Add runtime workspaces and explicit headroom, then calculate the minimum valid shard count: minimum GPUs per replica = ceil(total required bytes for one replica ÷ usable bytes per GPU). Treat this as a lower bound only; the actual partition must fit on its most memory-constrained rank or pipeline stage.

A nominal 70B-parameter checkpoint stored in BF16 requires about 140 GB, or roughly 130 GiB, for weights alone, so it will not fit on an 80 GB GPU. That total is nominally below the H200’s advertised 141 GB capacity, but actual fit depends on the model’s exact parameter count and usable device memory, and it leaves limited room for runtime workspaces, allocator overhead, and the KV cache. Raw weight capacity alone does not establish a viable one-GPU configuration. An 8-bit weight representation is approximately 70 GB before scales and metadata; an ideal 4-bit representation is approximately 35 GB before quantization overhead. Kernel support, accuracy, and the runtime’s actual packed format must be validated separately.

Once the memory budget is known, choose the simplest parallelism topology that fits and then benchmark it. vLLM’s parallelism and scaling guidance uses the same progression: a single GPU when the model and cache fit; tensor parallelism within a node when they do not; and tensor plus pipeline parallelism when the model exceeds the memory of one node. Each additional shard adds communication and synchronization, so memory fit and performance must be tested together.

HBM capacity and bandwidth establish different feasibility and performance envelopes across current accelerator generations:

H200 SXM and H200 NVL have the same listed HBM capacity and bandwidth, but their power envelopes, form factors, and system interconnects differ. Do not select a GPU from HBM capacity alone. Capacity determines whether the state can fit. Bandwidth, tensor-core throughput at the actual dtype, interconnect topology, kernel support, and software maturity determine how quickly the workload runs. The most defensible choice comes from a representative benchmark. 

3. Benchmark small before you scale

Once the workload fits, do not extrapolate directly from peak TFLOPS. Vendor figures may refer to different dtypes, sparsity assumptions, and boost conditions, and they exclude the host, network, storage, and framework overhead that shapes end-to-end performance. Benchmark the actual model, tensor shapes, precision policy, sequence distribution, batching policy, software stack, and data path.

Use a controlled benchmark protocol:

1. Lock the environment. Hold the container, driver, CUDA toolkit, framework, NCCL, attention kernels, compiler settings, tokenizer, and data pipeline constant across hardware and scale tests.

2. Measure steady state. Warm kernels and caches, separate graph compilation and model initialization from the timed region, and run enough steps or requests to observe variance, thermal stability, and stragglers.

3. Record workload and system metrics. For training, capture step time, tokens per second per GPU, peak allocated and reserved HBM, model FLOPS utilization, communication time, data stalls, and checkpoint I/O. For inference, capture request and token throughput together with p50, p95, and p99 time to first token, inter-token latency, end-to-end latency, KV-cache occupancy, queue depth, and preemptions.

4. Scale one variable at a time. Test the smallest valid GPU count, then increase within the same node before crossing node boundaries. Hold the global workload and quality target constant for strong-scaling tests; otherwise a larger batch or changed sequence mix can make the comparison meaningless.

The purpose of this instrumentation is diagnosis. Low throughput can be compute-bound, HBM-bandwidth-bound, communication-bound, host-bound, or input-I/O-bound, and each bottleneck requires a different fix. High reported GPU utilization does not prove useful work is being completed efficiently. NVIDIA’s Megatron Bridge logging guidance exposes iteration timing, memory statistics, world size, energy use, and rank-level timers so the bottleneck can be localized rather than guessed.

4. How many GPUs for inference? 

Inference sizing contains two different calculations: the GPU count inside one model replica, and the number of replicas required to meet traffic and reliability targets. Combining them hides whether the system is constrained by model fit, per-request latency, throughput, or redundancy.

Stage A: how many GPUs per replica?

• Start with one GPU only if the weights, KV cache at the intended concurrency, execution workspaces, and memory headroom all fit.

• Use tensor parallelism within a high-bandwidth node when the model does not fit on one GPU or when a single request needs more compute to meet latency. Measure the communication penalty; additional tensor-parallel ranks do not guarantee lower latency.

• Use pipeline parallelism, or a combined tensor-and-pipeline topology, when the model exceeds the usable HBM of one node. Account for stage imbalance, pipeline bubbles, and inter-node link bandwidth.

• If one replica already meets latency, prefer independent data-parallel replicas for horizontal throughput scaling. Retest the complete request distribution after every topology change.

Stage B: how many replicas?

Load test a production-equivalent replica using the expected prompt-length, output-length, arrival-rate, and burst distributions. Measure the maximum sustained throughput that still satisfies the tail-latency objective, not the throughput observed after the server is saturated.

Set target utilization below 1 because queueing delay rises nonlinearly near saturation, then add explicit failure or maintenance capacity, such as N+1 or a defined replica margin. Separate prefill and decode in the results: long prompts increase prefill compute and time to first token, while autoregressive decode is often constrained by memory bandwidth and KV-cache traffic. vLLM reports GPU KV-cache capacity and estimated maximum concurrency at startup. When cache pressure forces request preemption, vLLM may recompute evicted work, increasing tail latency; its optimization guidance describes the signal and mitigation options.

5. How long do you need the capacity? Persistent vs. temporary demand

Once the technical configuration is known, determine how long each exact capacity block is required. Duration is a separate decision from GPU count, but it determines whether that count should be held continuously or acquired only for a bounded run.

Production inference, scheduled pipelines, and continuous training can create a persistent floor. Experiments, evaluation campaigns, release launches, and large training runs create temporary demand above it. Measure these curves separately by GPU type, node topology, and software environment; an H100 GPU-hour is not automatically interchangeable with an H200 GPU-hour, much less a multi-node InfiniBand allocation.

For each like-for-like capacity block, calculate duty cycle from observed demand:

This break-even condition assumes equivalent hardware, availability, billing granularity, storage, networking, support, and contract terms, with no productive use for idle reserved hours. If capacity can be reassigned to another workload, include that utilization. If access itself has operational value, quantify the cost of a missed deadline or unavailable GPU rather than treating hourly rate as the only variable.

Hyperbolic supports both models: On-Demand for usage-based capacity and Reserved for prepaid fixed-term capacity. Start On-Demand while model behavior, topology, and demand remain uncertain. Move the measured baseline into Reserved only after the configuration and sustained duty cycle are stable, while keeping short-lived or volatile workloads on On-Demand capacity. 

A worked example: one team, two workloads

Consider an AI team with two independently benchmarked workloads. These values illustrate the calculation and are not Hyperbolic pricing or performance claims.

Let rᵣ be the reserved hourly rate and rₒ the on-demand hourly rate. Using 730 hours as an average planning month:

Provisioning all 10 GPUs continuously would commit 7,300 GPU-hours per month. The measured workloads use 2,228 GPU-hours, so fleet utilization would be about 30.5% if the remaining hours could not be reassigned. The 8-GPU training block alone has a 96 ÷ 730 ≈ 13.2% monthly duty cycle. Splitting the persistent 2-GPU floor from the 8-GPU burst avoids converting a four-day training requirement into a month-long commitment.

Six mistakes that inflate GPU bills

These are the sizing errors that most often produce invalid benchmarks or unnecessary GPU-hours:

• Sizing from parameter count alone. Parameter count approximates weight storage, not activations, optimizer state, KV cache, token volume, kernel efficiency, or communication.

• Treating aggregate HBM as pooled memory. Every rank and pipeline stage must fit locally after replication, sharding, temporary workspaces, and headroom are included.

• Assuming linear scaling. Doubling GPU count does not halve runtime when collectives, pipeline bubbles, synchronization, stragglers, and I/O consume a growing share of the step.

• Benchmarking inference at saturation. Maximum throughput is irrelevant if queueing pushes p95 or p99 latency beyond the service objective.

• Optimizing for GPU-hour price instead of completed-workload economics. Compare cost per checkpoint, evaluated sample, or output token at the required quality and latency.

• Turning a peak into the baseline. A two-week burst should not become a twelve-month commitment unless the remaining capacity has a measured workload. 

The GPU capacity planning checklist

Deploy the right GPU capacity for your workload

Choose from H100, H200, and B200 GPUs with capacity options designed for both flexible experimentation and sustained production demand.

Deploy GPUs →

Frequently asked questions