> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hyperbolic.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Verifying Instance Performance

> Benchmark tests to confirm your GPUs, interconnect, and storage are performing as expected

Use these checks after launching an instance to confirm you're getting the performance you're paying for, and before opening a support ticket about slow workloads. For the hardware specs to compare against (memory, bandwidth, interconnect), see the [GPU comparison table](/docs/overview/platform-comparison#available-gpus).

## Quick health check

```bash theme={null}
# GPUs visible, correct model and count, driver loaded
nvidia-smi

# Detailed per-GPU state: clocks, power limits, ECC errors
nvidia-smi -q
```

Confirm that:

* The **GPU model and count** match what you rented (H100 80GB, H200 141GB, or B200 192GB)
* **ECC error counts** are zero or not climbing
* No GPU is stuck at low clocks or in a fallback power state while idle

## GPU diagnostics (DCGM)

NVIDIA's Data Center GPU Manager runs structured hardware diagnostics:

```bash theme={null}
# Install if not present (Ubuntu)
sudo apt-get install -y datacenter-gpu-manager
sudo systemctl start nvidia-dcgm

# Level 2 diagnostic (~2 minutes): PCIe, memory, compute sanity
dcgmi diag -r 2

# Level 3 (~30 minutes): adds stress and stability tests
dcgmi diag -r 3
```

Any test reported as **Fail** is grounds for a support ticket — include the full output.

## Compute throughput

A large matrix multiplication measures real achievable throughput:

```python theme={null}
import torch, time

n = 8192
a = torch.randn(n, n, dtype=torch.bfloat16, device="cuda")
b = torch.randn(n, n, dtype=torch.bfloat16, device="cuda")

for _ in range(10):  # warmup
    a @ b
torch.cuda.synchronize()

iters = 100
start = time.time()
for _ in range(iters):
    a @ b
torch.cuda.synchronize()
elapsed = time.time() - start

tflops = (2 * n**3 * iters) / elapsed / 1e12
print(f"{tflops:.0f} TFLOPS (BF16)")
```

Compare against the GPU's datasheet peak for dense BF16. Well-tuned large matmuls should reach a substantial fraction of peak; results dramatically below that across repeated runs (rule out thermal throttling and shared load first) are worth reporting to support.

## Memory bandwidth

```bash theme={null}
git clone https://github.com/NVIDIA/nvbandwidth && cd nvbandwidth
sudo apt-get install -y libboost-program-options-dev
cmake . && make

# Host-to-device, device-to-host, and device-to-device bandwidth
./nvbandwidth
```

Device-to-device results should approach the memory bandwidth listed for your GPU (3.35 TB/s H100, 4.8 TB/s H200, 8 TB/s B200).

## Multi-GPU and interconnect

For multi-GPU instances and InfiniBand clusters, verify collective bandwidth with [nccl-tests](https://github.com/NVIDIA/nccl-tests):

```bash theme={null}
git clone https://github.com/NVIDIA/nccl-tests && cd nccl-tests
make

# All-reduce across all 8 GPUs, 8 B to 8 GB message sizes
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8
```

Check the **busbw** column at large message sizes:

* On NVLink-connected nodes, bus bandwidth should scale toward the NVLink spec (900 GB/s on H100/H200, 1.8 TB/s on B200) — several hundred GB/s at large sizes is healthy on 8x H100.
* On InfiniBand clusters, run the multi-node variant (via MPI) and confirm inter-node bandwidth is consistent with the cluster's InfiniBand configuration (up to 3.2 Tb/s).

You can also confirm the interconnect topology directly:

```bash theme={null}
nvidia-smi topo -m   # NV-links between GPU pairs
ibstat               # InfiniBand link state and rate (clusters)
```

## Storage I/O

```bash theme={null}
sudo apt-get install -y fio

# Sequential write/read throughput on the current volume
fio --name=seqwrite --rw=write --bs=1M --size=4G --numjobs=1 --direct=1
fio --name=seqread  --rw=read  --bs=1M --size=4G --numjobs=1 --direct=1
```

See [Storage and Ports](/docs/on-demand/storage-and-ports) for the difference between local NVMe and network volumes — expected throughput differs substantially between the two.

## If results are below expectations

1. Re-run the failing test at least twice to rule out transient load.
2. Capture the output of `nvidia-smi -q` alongside the benchmark output.
3. Contact [support](/docs/general/support) with the instance ID, region, and the collected outputs.