> ## Documentation Index
> Fetch the complete documentation index at: https://hyperbolic.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Managing Instances

> Create, monitor, and manage high-performance GPU instances through the Web UI

Master the full lifecycle of GPU instance management on Hyperbolic - from creation to termination, including monitoring, scaling, and troubleshooting.

## Creating Instances

### Web UI Method

**1. Navigate to On-Demand GPU platform**

Go to [app.hyperbolic.ai/gpus](https://app.hyperbolic.ai/gpus) and browse available GPUs.

**2. Select GPU Configuration**

* Choose GPU type (H100 80GB, H200 141GB, B200 192GB, B300 288GB — availability varies by region)
* Select quantity
* Pick region for optimal latency
* Choose InfiniBand if needed for multi-GPU

**3. Configure Instance**

* **Storage**: Configure as needed for your storage needs
* **Label**: Name your instance for easy identification
* **SSH Key** (required): select a saved SSH key, or paste a new one. The instance is provisioned with the key you select. Some regions let you attach more than one key. See [Attach multiple keys to an instance](https://hyperbolic.ai/docs/general/account-management.md#attach-multiple-keys-to-an-instance). Manage saved keys under [Settings > SSH Keys](https://app.hyperbolic.ai/settings/ssh-keys). Under an [Organization](https://hyperbolic.ai/docs/general/organizations.md#ssh-keys), only Organization keys can be selected.

**4. Launch**

Review pricing and click "Start Building". Instance will be ready in under 5 minutes.

## Connecting to Instances

### SSH Connection

Basic SSH connection:

```bash theme={null}
# Standard connection
ssh ubuntu@<instance-ip>

# With specific key
ssh -i ~/.ssh/hyperbolic_key ubuntu@<instance-ip>
```

### File Transfer

> [tab] SCP
>
> ```bash theme={null}
> # Upload file
> scp model.pth ubuntu@<instance-ip>:/home/ubuntu/
>
> # Download file
> scp ubuntu@<instance-ip>:/home/ubuntu/results.csv ./
>
> # Upload directory
> scp -r dataset/ ubuntu@<instance-ip>:/home/ubuntu/
> ```

> [tab] Rsync
>
> ```bash theme={null}
> # Sync directory (upload)
> rsync -avz --progress dataset/ ubuntu@<instance-ip>:/home/ubuntu/dataset/
>
> # Sync with deletion of removed files
> rsync -avz --delete --progress local/ ubuntu@<instance-ip>:/home/ubuntu/remote/
>
> # Resume interrupted transfer
> rsync -avz --partial --progress large_file.tar ubuntu@<instance-ip>:/data/
> ```

> [tab] SFTP
>
> ```bash theme={null}
> # Interactive SFTP session
> sftp ubuntu@<instance-ip>
>
> # SFTP commands
> sftp> put model.pth
> sftp> get results/
> sftp> ls
> sftp> pwd
> sftp> exit
> ```

## Instance Lifecycle Management

### Rebooting an instance

Where supported, each node has a **Reboot** button in the **Instance Management** section of the instance's details page. Use it instead of `sudo reboot` from inside the instance — a reboot triggered from the console goes through the platform, which keeps the instance's state and network configuration intact.

While a node reboots, its status shows **Rebooting**. On a multi-node instance, the instance itself stays **active** and its other nodes keep running.

Depending on the instance, a reboot is one of two kinds:

* **Warm reset.** Most virtual machines get a warm, in-place reset of the guest OS.
* **Hard reboot.** Bare-metal servers and some virtual machines are power-cycled, the same as cutting and restoring power. The guest OS can't block a hard reboot. A bare-metal reboot can take up to about 20 minutes.

Either way, your disk contents are preserved, but anything held only in memory is lost. Save your work and stop running jobs before you reboot. The node is unreachable over SSH until the reboot finishes. You can't start a second reboot on a node while one is in progress.

To reboot through the API, send `POST /v2/on-demand/rentals/reboot-instance` with the rental ID and the node's `instanceRef`. Only reboot nodes whose `instances[].supportsReboot` is `true`. See [Reboot a single instance within a rental](https://hyperbolic.ai/docs/api-reference/rentals/reboot-a-single-instance-within-a-rental.md).

A node's `instanceRef` has the form `<rentalId>-<node index>`, with node indexes starting at `0`. Take the value from the rental's `instances[].instanceRef` in the API, or copy it from beside the node's name (such as **Node 0**) in the **Instance Management** section. An `instanceRef` that doesn't belong to the rental returns `INVALID_INSTANCE_REF`.

```bash theme={null}
curl -X POST https://api.hyperbolic.ai/v2/on-demand/rentals/reboot-instance \
  -H "Authorization: Bearer $HYPERBOLIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"rentalId": "<rental-id>", "instanceRef": "<rental-id>-0"}'
```

If a node has no **Reboot** button, that machine type doesn't support platform reboots. Contact support rather than rebooting from inside the OS; on some machines an in-OS reboot can leave the instance unrecoverable.

Onboard storage is preserved across a platform reboot. It is **not** preserved across reprovisioning or termination — see [Storage and Ports](https://hyperbolic.ai/docs/on-demand/storage-and-ports.md).

> [!WARNING]
>
> Rebooting because of a hardware error — an Xid in `dmesg`, a GPU that has dropped off the bus, a row-remap or SRMA message? Back up first. A reboot on a faulted host doesn't always come back cleanly, and the recovery path is sometimes a fresh instance, which onboard data doesn't survive.

### Terminating Instances

> [!WARNING]
>
> Termination is permanent. All data on the instance will be lost. Always back up important data before terminating.

To terminate an instance:

1. Go to "My Instances" in the Web UI.
2. Open the **⋮** menu on the instance's card, or open the instance and select **Actions** on its details page.
3. Select **Terminate**.
4. Confirm the deletion.

## Monitoring and Metrics

### Real-Time GPU Monitoring

SSH into your instance and use these commands:

```bash theme={null}
# Basic GPU status
nvidia-smi

# Continuous monitoring (updates every 1 second)
watch -n 1 nvidia-smi

# Detailed GPU metrics
nvidia-smi -q

# Show running processes
nvidia-smi pmon

# GPU utilization over time
nvidia-smi dmon
```

### System Monitoring

```bash theme={null}
# System resources
htop

# Disk usage
df -h

# Memory usage
free -h

# Network statistics
nethogs  # Install with: sudo apt install nethogs

# Process monitoring
ps aux | grep python
```

### Setting Up Custom Monitoring

> [tab] Prometheus + Grafana
>
> ```bash theme={null}
> # Install DCGM exporter for GPU metrics
> docker run -d --gpus all --rm -p 9400:9400 nvidia/dcgm-exporter:latest
>
> # Install Prometheus
> docker run -d -p 9090:9090 \
>   -v prometheus.yml:/etc/prometheus/prometheus.yml \
>   prom/prometheus
>
> # Install Grafana
> docker run -d -p 3000:3000 grafana/grafana
> ```

> [tab] Weights & Biases
>
> ```python theme={null}
> import wandb
> import torch
>
> # Initialize W&B
> wandb.init(project="hyperbolic-training")
>
> # Log GPU metrics automatically
> wandb.watch(model)
>
> # Custom GPU logging
> for epoch in range(num_epochs):
>     gpu_utilization = torch.cuda.utilization()
>     gpu_memory = torch.cuda.memory_allocated() / 1024**3
>
>     wandb.log({
>         "gpu_utilization": gpu_utilization,
>         "gpu_memory_gb": gpu_memory,
>         "epoch": epoch
>     })
> ```

## Managing Multiple Instances

### Viewing All Instances

Go to [app.hyperbolic.ai/instances](https://app.hyperbolic.ai/instances) to view all your instances with:

* Instance status
* GPU type and configuration
* Region
* Running time and costs
* Quick action buttons

### Batch Operations

**Parallel SSH Commands**:

```bash theme={null}
# Run command on multiple instances
for ip in 192.168.1.10 192.168.1.11 192.168.1.12; do
    ssh ubuntu@$ip "nvidia-smi" &
done
wait

# Using GNU parallel
parallel -j 4 ssh ubuntu@{} "python train.py" ::: \
  instance1.hyperbolic.ai \
  instance2.hyperbolic.ai \
  instance3.hyperbolic.ai
```

**Using Ansible**:

```yaml theme={null}
# inventory.yml
all:
  hosts:
    h100-1:
      ansible_host: 192.168.1.10
    h100-2:
      ansible_host: 192.168.1.11
    h100-3:
      ansible_host: 192.168.1.12
  vars:
    ansible_user: ubuntu
    ansible_ssh_private_key_file: ~/.ssh/hyperbolic_key

# Run command on all instances
ansible all -i inventory.yml -m shell -a "nvidia-smi"
```

## Instance States and Troubleshooting

### Instance States

| State | Description | Billing |
| - | - | - |
| **Pending** | Instance is being provisioned | No |
| **Running** | Instance is ready and accessible — billing starts here | Yes |
| **Starting** | Instance is booting up | No |
| **Converting** | Reservation has ended; converting to on-demand billing (or terminating) within \~10 minutes | Yes |
| **Terminating** | Instance is being deleted | No |
| **Failed** | Instance failed to start | No |

### Common Issues and Solutions

> [accordion] Instance stuck in 'Pending' state
>
> **Solution**:
>
> * Allow a few minutes for provisioning
> * Check region availability in the dashboard
> * Contact [support](mailto:support@hyperbolic.ai) if pending > 30 minutes
>
> Check the instance status in your dashboard for updates.

> [accordion] Cannot SSH to instance
>
> **Solution**:
>
> * Verify instance is in "Running" state
> * Check SSH key is authorized
> * Confirm IP address is correct
> * Test network connectivity
>
> ```bash theme={null}
> # Debug SSH connection
> ssh -vvv ubuntu@<instance-ip>
> ```

> [accordion] GPU not available in instance
>
> **Solution**:
>
> * Verify GPU driver is loaded
> * Check Docker runtime configuration
> * Restart instance if needed
>
> ```bash theme={null}
> # Check GPU availability
> nvidia-smi
>
> # Check driver
> nvidia-smi -q | grep "Driver Version"
>
> # For Docker containers
> docker run --gpus all nvidia/cuda:12.2.0-base nvidia-smi
> ```

> [accordion] Instance automatically terminated
>
> **Possible causes**:
>
> * Insufficient account balance
> * Violation of terms of service
> * Hardware failure (rare)
>
> **Solution**:
>
> * Check billing status in your dashboard
> * Review instance logs
> * Contact support for clarification

> [!NOTE]
>
> Still having issues? Contact [**support@hyperbolic.ai**](mailto:support@hyperbolic.ai) or use the chat in the bottom-right corner of the [app](https://app.hyperbolic.ai) after you sign in.

## Best Practices

> [card] Use Instance Labels
>
> Name your instances clearly to track different projects and experiments.

> [card] Implement Auto-shutdown
>
> Set up scripts to automatically terminate idle instances to avoid unnecessary charges.

> [card] Regular Backups
>
> Schedule regular backups of important data to external storage (S3, GCS, etc.).

> [card] Monitor Costs
>
> Set up billing alerts and regularly review instance usage to optimize costs.

### Auto-shutdown Script Example

```python theme={null}
#!/usr/bin/env python3
import subprocess
import time
import sys

def get_gpu_utilization():
    """Get current GPU utilization percentage"""
    result = subprocess.run(
        ['nvidia-smi', '--query-gpu=utilization.gpu', '--format=csv,noheader,nounits'],
        capture_output=True,
        text=True
    )
    return int(result.stdout.strip())

def auto_shutdown(idle_minutes=30, threshold=5):
    """Shutdown instance if GPU idle for specified minutes"""
    idle_count = 0
    check_interval = 60  # Check every minute

    while True:
        util = get_gpu_utilization()

        if util < threshold:
            idle_count += 1
            print(f"GPU idle ({util}%). Idle count: {idle_count}/{idle_minutes}")

            if idle_count >= idle_minutes:
                print("Shutting down due to inactivity...")
                subprocess.run(['sudo', 'shutdown', '-h', 'now'])
                sys.exit(0)
        else:
            idle_count = 0
            print(f"GPU active ({util}%). Reset idle counter.")

        time.sleep(check_interval)

if __name__ == "__main__":
    auto_shutdown(idle_minutes=30, threshold=5)
```

## Advanced Configuration

### Custom Startup Scripts

Create a startup script that runs when instance boots:

```bash theme={null}
#!/bin/bash
# /home/ubuntu/startup.sh

# Mount additional storage
sudo mount /dev/nvme1n1 /data

# Start Jupyter
jupyter notebook --no-browser --port=8888 &

# Start TensorBoard
tensorboard --logdir=/home/ubuntu/logs --port=6006 &

# Start monitoring
python /home/ubuntu/monitor_gpu.py &
```

### Data Persistence Strategies

> [tab] Cloud Storage
>
> ```bash theme={null}
> # Install AWS CLI
> sudo apt-get update
> sudo apt-get install awscli -y
>
> # Configure AWS credentials
> aws configure
>
> # Sync data to S3
> aws s3 sync /home/ubuntu/models/ s3://my-bucket/models/
>
> # Download from S3
> aws s3 sync s3://my-bucket/dataset/ /home/ubuntu/dataset/
> ```

> [tab] Git LFS
>
> ```bash theme={null}
> # Install Git LFS
> curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
> sudo apt-get install git-lfs
>
> # Initialize Git LFS
> git lfs install
>
> # Track large files
> git lfs track "*.pth"
> git lfs track "*.h5"
>
> # Push to repository
> git add .
> git commit -m "Save model checkpoint"
> git push
> ```

## Next Steps

> [card] [Storage & Ports](https://hyperbolic.ai/docs/on-demand/storage-and-ports.md)
>
> Configure persistent storage and network settings


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.