GPU Monitoring for ML: SM Efficiency, Memory Bandwidth, and Bottlenecks

Your training job just crashed. Again. The error message mentions memory, but system monitors show plenty of RAM available. After hours of debugging, the culprit emerges: GPU memory exhaustion that went undetected because nobody was actively tracking utilization metrics. For developers and researchers deploying AI models, this scenario is frustratingly common and entirely preventable.

According to "The State of AI Infrastructure at Scale 2024," over 75% of organizations report GPU utilization below 70% at peak load. This means that even as demand for AI capacity accelerates, the majority of one of the most valuable computing resources sits idle. Learning how to check GPU usage effectively transforms this wasted potential into productive compute time.

Why Monitoring GPU Usage Matters for AI Projects

GPU resources represent significant investments, whether in purchased hardware or cloud computing costs. Without proper monitoring, teams operate blind to critical bottlenecks, inefficiencies, and resource waste that directly impact project timelines and budgets.

Effective monitoring serves multiple purposes beyond simple resource tracking. It identifies performance bottlenecks before they cause failures, validates that expensive hardware actually delivers value, enables capacity planning for scaling projects, and provides data for cost optimization decisions. 

When teams know how to monitor GPU usage properly, they gain visibility into the complete picture of their computational infrastructure.

Basic Tools to Check GPU Usage

Several straightforward tools provide immediate insights into GPU performance without requiring complex setup or specialized knowledge.

NVIDIA-SMI Command Line Tool

The nvidia-smi utility ships with NVIDIA drivers and offers the quickest method to check GPU usage on systems running NVIDIA hardware. Running this command displays real-time statistics, including GPU utilization percentage, memory consumption, temperature readings, and power draw.

To check GPU usage with nvidia-smi, open a terminal and type:

nvidia-smi

The output shows utilization metrics for each installed GPU. For continuous monitoring, add a refresh interval:

nvidia-smi -l 1

This updates statistics every second, providing a live view of GPU activity during training or inference operations.

GPU Stat for Cleaner Output

While nvidia-smi provides comprehensive information, its output can be cluttered for quick checks. The gpustat tool offers a more user-friendly alternative with concise, color-coded summaries.

Install gpustat using pip:

pip install gpustat

Running gpustat displays GPU index, name, temperature, utilization percentage, memory usage, and running processes in a compact format that's easier to parse at a glance.

Framework-Specific Monitoring

PyTorch and TensorFlow include built-in functions to query GPU status programmatically within training scripts. These framework tools enable dynamic monitoring that can trigger actions based on utilization thresholds.

PyTorch provides torch.cuda.memory_allocated() and torch.cuda.utilization() for checking memory and compute usage, respectively. TensorFlow offers similar functionality through tf.config.experimental.get_memory_info().

Advanced GPU Usage Monitor Solutions

For production environments and complex multi-GPU setups, advanced monitoring tools provide deeper insights and historical tracking capabilities.

NVIDIA Nsight Systems

NVIDIA Nsight Systems delivers professional-grade profiling for deep learning workloads. This tool visualizes GPU and CPU activities over time, highlighting bottlenecks and inefficiencies that simpler tools miss.

The timeline view shows exactly when GPUs sit idle versus actively computing, revealing patterns that indicate data loading bottlenecks, synchronization issues, or suboptimal batch sizes. Kernel execution times expose which operations consume most resources, guiding optimization efforts.

Prometheus and Grafana Stack

Organizations running multiple GPU servers benefit from centralized monitoring using Prometheus for metrics collection and Grafana for visualization. This combination provides dashboards showing utilization trends across entire GPU clusters.

The NVIDIA GPU exporter for Prometheus scrapes metrics from nvidia-smi and makes them available for long-term storage and analysis. Custom alerts notify teams when utilization drops below thresholds or when memory approaches capacity limits.

Cloud Platform Tools

Cloud providers offer native monitoring solutions integrated with their GPU instances. AWS CloudWatch tracks GPU metrics for EC2 instances with attached accelerators. Google Cloud Monitoring provides similar capabilities for Compute Engine VMs with GPUs. Azure Monitor covers GPU metrics for Azure VMs.

These platform-specific tools integrate seamlessly with other cloud services, enabling unified monitoring of compute, storage, and network resources alongside GPU metrics.

Key Metrics to Track When Monitoring

Knowing how to check GPU usage requires recognizing which metrics actually matter for AI workloads and what their values indicate about system health.

GPU Utilization Percentage

This metric shows the percentage of time one or more kernels executed on the GPU. While commonly tracked, GPU utilization can be misleading. A GPU showing 100% utilization might still perform far below the theoretical maximum if kernels fail to fully occupy available compute cores.

Memory Utilization

Memory metrics reveal both allocated memory and actual usage. Training large models requires substantial VRAM, but memory leaks or inefficient implementations can waste this limited resource. Tracking memory over time identifies gradual increases that signal leaks requiring investigation.

Streaming Multiprocessor Efficiency

SM efficiency, also called SM activity, measures the percentage of streaming multiprocessors that actively process work during kernel execution. Low SM efficiency despite high GPU utilization indicates poorly parallelized code that fails to leverage the GPU's architecture effectively.

Temperature and Power Draw

Thermal readings ensure GPUs operate within safe limits. Sustained high temperatures trigger throttling that reduces performance. Power draw provides another efficiency indicator, as GPUs running at full capacity consume near their thermal design power rating.

Common GPU Usage Patterns in AI Workloads

Different AI tasks create distinct utilization signatures that help diagnose issues and validate proper operation.

Training Workloads

Model training typically maintains high, steady GPU utilization during forward and backward passes. Periodic dips correspond to data loading between batches. If utilization frequently drops to zero, data pipelines likely bottleneck training speed.

Well-optimized training achieves 70-85% average utilization, accounting for data loading overhead. Utilization consistently below 60% suggests opportunities for improvement through larger batch sizes, mixed precision training, or optimized data loaders.

Inference Operations

Inference creates different patterns depending on the deployment mode. Batch inference processes multiple examples simultaneously, maintaining moderate utilization during processing with idle periods between batches. Real-time inference shows brief utilization spikes when serving predictions, with substantial idle time waiting for requests.

Multi-GPU Setups

Distributed training across multiple GPUs should show similar utilization across all devices. Significant imbalance between GPUs indicates poor workload distribution, inefficient communication patterns, or model parallelism issues requiring investigation.

Optimizing Based on Usage Data

Monitoring reveals problems, but optimization requires action based on discovered patterns. Several strategies address common issues identified through GPU usage monitoring.

Addressing Low Utilization

When monitoring shows utilization consistently below targets, several factors might be responsible:

• CPU bottlenecks: Data preprocessing on the CPU cannot keep pace with GPU consumption

• Small batch sizes: Insufficient work to fully occupy GPU cores

• I/O limitations: Slow disk or network reading data for training

• Synchronization overhead: Frequent communication between GPUs in distributed setups

Solutions include increasing dataloader workers, implementing data prefetching, using mixed precision to increase batch sizes, and profiling to identify specific bottlenecks.

Managing Memory Constraints

Memory utilization near maximum capacity risks out-of-memory errors that terminate training. Gradient accumulation simulates larger batch sizes across multiple smaller batches, trading throughput for memory savings. Mixed precision training reduces memory footprint by using FP16 for most operations while maintaining FP32 for critical calculations.

Gradient checkpointing trades computation for memory by recomputing intermediate activations during backpropagation rather than storing them. This technique enables training larger models on available hardware at the cost of increased training time.

Improving SM Efficiency

Low SM efficiency indicates kernels that fail to fully utilize the GPU architecture. Flash Attention and other optimized attention mechanisms replace memory-bound operations with more compute-efficient implementations. Kernel fusion combines multiple operations into a single kernel, reducing memory traffic and improving parallelization.

Monitoring in Production Environments

Production AI systems require a robust monitoring infrastructure that goes beyond ad-hoc checks during development.

Automated alerting notifies teams when metrics drift outside acceptable ranges. Setting up alerts for utilization drops, memory exhaustion, temperature spikes, and power anomalies enables rapid response to issues before they impact users.

Historical trend analysis reveals patterns over time. Gradual utilization decreases might indicate model drift or data distribution changes. Memory usage creep suggests potential leaks requiring investigation. Comparing current metrics against historical baselines helps distinguish normal variation from genuine problems.

Integration with broader observability platforms provides context for GPU metrics. Correlating GPU utilization with application performance metrics, error rates, and user experience data creates a complete operational picture that guides improvement priorities.

Best Practices for Regular Monitoring

Consistent monitoring habits prevent problems and optimize resource utilization over time. Establishing regular review cycles ensures GPU infrastructure receives appropriate attention.

• During development, check GPU usage frequently while tuning hyperparameters and architectural choices. Quick iteration with different batch sizes, learning rates, and model configurations requires immediate feedback on resource implications.

• In production, automated monitoring replaces manual checks. Setting up dashboards that update continuously allows teams to spot issues without constant attention. Weekly or monthly reviews of historical trends inform capacity planning and identify optimization opportunities.

• For cost optimization, correlating GPU usage with cloud billing or energy costs quantifies the financial impact of inefficiencies. Unutilized GPUs represent wasted money, whether from idle cloud instances or underutilized owned hardware consuming electricity without delivering value.

Conclusion

Knowing how to check GPU usage effectively separates successful AI projects from those plagued by mysterious failures, cost overruns, and performance problems. Basic tools like nvidia-smi provide immediate visibility for quick checks during development. Advanced solutions like Nsight Systems and centralized monitoring stacks support production deployments at scale.

The key lies not just in collecting metrics but in interpreting them correctly and taking action based on insights. GPU utilization alone can mislead, requiring deeper metrics like SM efficiency and memory patterns for accurate assessment. Regular monitoring combined with systematic optimization based on discovered patterns transforms expensive GPU resources into productive computational assets that accelerate AI innovation while controlling costs.

For developers, researchers, and startups building AI solutions, investing time to master GPU usage monitoring pays dividends throughout project lifecycles. The visibility gained prevents costly surprises, enables data-driven optimization decisions, and ensures that computational investments deliver maximum value toward achieving AI objectives.

About Hyperbolic

Hyperbolic is the on-demand AI cloud made for developers. We provide fast, affordable access to compute, inference, and AI services. Over 195,000 developers use Hyperbolic to train, fine-tune, and deploy models at scale.

Our platform has quickly become a favorite among AI researchers, including those like Andrej Karpathy. We collaborate with teams at Hugging Face, Vercel, Quora, Chatbot Arena, LMSYS, OpenRouter, Black Forest Labs, Stanford, Berkeley, and beyond.

Founded by AI researchers from UC Berkeley and the University of Washington, Hyperbolic is built for the next wave of AI innovation—open, accessible, and developer-first.

Website | X | Discord | LinkedIn | YouTube | GitHub | Documentation