Today, we’re introducing Flux, the next generation of Hyperbolic inference, built on Forge, our unified control plane for orchestration across clouds and clusters, dynamic GPU allocation, and virtualization.

We launched Hyperbolic’s inference service in early 2024 to make open models easier to access and use. Since then, it has helped attract more than 250,000 developers to our platform. In his Deep Dive into LLMs like ChatGPT, Andrej Karpathy described Hyperbolic as his favorite place to interact with base models.

That experience taught us that efficient inference takes more than an optimized serving engine. It also depends on how well we orchestrate workloads across the underlying compute, matching them to available GPU capacity as demand changes.

That’s why we built Forge, our unified control plane for integrating and virtualizing compute across heterogeneous infrastructure. Forge lets us orchestrate workloads across clouds and clusters and dynamically allocate GPU capacity.

Building on that foundation, we created Flux to bring inference onto the same control plane as our other compute services. Running Flux on Forge is designed to let us shift GPU capacity between workloads as demand changes, putting otherwise idle GPUs to work across the platform. The goal is higher utilization and more revenue per GPU-hour across our fleet.

With Flux, developers can access a wide range of models, including GLM-5.3-Flash, through one OpenAI-compatible API. Build AI applications and stream responses without provisioning GPUs or operating model-serving infrastructure. Unified authentication, token metering, and token-based billing are handled through your Hyperbolic account.

What production LLM inference requires

Every inference request crosses several systems: account authentication, request routing, GPU execution, streaming, and usage settlement. Each boundary needs clear access controls, traceable accounting, and operational visibility.

We designed Flux around four requirements:

  • Observable performance. Time to first token and inter-token latency show how a response behaves. Queue depth, throughput, cache utilization, and GPU health help explain why. Billing settlement stays outside the token-generation loop.

  • Usage accounting for streaming responses. Flux separates usage capture from charge settlement, so billing runs outside the response-streaming path.

  • Metering by token type. Input, cached input, and output tokens can have different prices, with reasoning tokens billed as output. Usage records need to preserve those distinctions where the model and serving runtime support them.

  • Secure access to distributed GPUs. Serving engines run on Forge-managed capacity behind a controlled ingress, with protected connections from the API edge.

How a request flows through Flux

Flux separates the public API, model serving, and billing into distinct responsibilities. The API edge authenticates requests, checks account balances, routes requests to the appropriate model backend, and meters token usage. Models hosted by Hyperbolic and models served through external providers are accessed through the same API.

  • Send a request. A client calls the OpenAI-compatible endpoint with a Hyperbolic API key. The interface supports Chat Completions and streaming responses.

  • Authenticate the account. The edge resolves the key to a user or organization and checks access through Hyperbolic’s account system. Identity and balance remain in the existing account infrastructure.

  • Route to a model backend. The edge selects a backend by model name. For Hyperbolic-hosted models, requests travel through the GPU ingress to an SGLang serving engine, with routing across replicas. External routes send requests to additional model-serving backends. Responses return through Hyperbolic’s edge in either case, with authentication and billing handled by Hyperbolic.

  • Stream the response. Generated tokens return through the edge to the client, while the gateway captures the request’s token usage.

  • Record usage and settle charges. Usage records pass to the billing system for asynchronous settlement, outside the response-streaming path.

This separation lets the serving infrastructure focus on processing requests while the account and billing systems manage access, prices, and charges.

Token metering and billing

Accurate inference billing starts with a record of what each request consumed. Flux separates token metering, usage delivery, and price calculation so each part can be inspected independently.

Flux’s gateway captures token counts by type, including input, cached input, and output tokens, with reasoning tokens counted as output. It does not calculate currency charges. Hyperbolic’s billing system maps those counts to the applicable model prices.

That boundary matters for both operations and product development. A pricing change does not require a change to the serving runtime. New model pricing can use the same usage-recording path. Cached tokens can be distinguished from uncached input instead of being grouped into one count.

The usage pipeline is designed to persist captured records before forwarding them to the billing service. Request identifiers allow the billing service to deduplicate repeated submissions. Settlement runs asynchronously, outside the response-streaming path.

Token metering and billing policy are separate concerns. The serving layer supplies usage counts; the billing system applies model prices and records charges against the customer’s balance. Usage returned by supported model backends flows through the same metering and billing path. This separation supports usage reconciliation without moving pricing logic into the serving runtime.

Inference security

For Hyperbolic-hosted models, clients access Flux through the API edge. Engine APIs, administrative interfaces, and metrics endpoints are reachable only through the mutually authenticated GPU ingress. Customer API keys are authenticated against Hyperbolic’s existing account system before requests reach a model backend. Externally routed models send requests to a third-party provider; the GPU ingress architecture described here applies to Hyperbolic-hosted capacity.

For Hyperbolic-hosted capacity, the architecture uses mutual TLS between the edge and GPU ingress to authenticate the connection and encrypt traffic between those components. Authentication at the public endpoint controls customer access; the protected internal connection controls access to Hyperbolic’s serving infrastructure.

Performance metrics track latency, throughput, queue depth, and cache usage without including prompt or completion text. Keeping these metrics separate from billing records helps the team investigate service performance.

Observability and performance monitoring

A successful HTTP response does not tell you whether inference is performing well. Flux’s observability design combines engine-level telemetry with public-endpoint checks to connect customer-facing latency with queueing, cache pressure, and GPU health. Engine-level signals apply to models hosted by Hyperbolic; external model routes also depend on provider performance and network latency.

The engine-level signals include:

  • Time to first token: how long a request waits before generation begins.

  • Inter-token latency: the interval between generated tokens during streaming.

  • Throughput and prompt processing: how much work each engine handles.

  • Queue depth: whether incoming demand is outpacing serving capacity.

  • KV-cache utilization and GPU health: whether memory pressure or infrastructure problems are constraining performance.

Synthetic requests are part of the public-endpoint monitoring design, alongside engine-level telemetry. Together, these signals help the team investigate prompt processing, queued requests, memory pressure, and unhealthy capacity, and size serving capacity by model. The current edge is in us-west-2. Requests from outside the US can incur additional network latency, and external model routes add provider hops. Responses stream back through the Hyperbolic edge rather than directly from the model backend to the client.

What this means for production teams

Flux brings three operational requirements into one inference service: controlled access to private serving infrastructure, token usage that can be reconciled with charges, and monitoring that connects response latency to engine and GPU performance.

Use familiar tools. An OpenAI-compatible API reduces integration work for applications built around Chat Completions. Configure the endpoint, API key, and supported model, then validate the features your application uses.

Manage inference and GPUs in one account. Flux connects to Hyperbolic’s existing identity and balance systems, bringing token-based usage and GPU capacity into the same platform.

Choose capacity for your workload. An inference API lets you call a model without operating the serving infrastructure yourself. Dedicated GPUs offer more control over deployment and utilization. For sustained workloads, the economics depend on model size, traffic, batching, and how effectively the GPUs are used.

Hyperbolic supports that progression through on-demand GPUs, reserved capacity, and Managed Kubernetes. Teams can evaluate dedicated infrastructure as their workloads grow.

Models available on the Flux inference API

Flux serves a wide range of models through one OpenAI-compatible API, with Hyperbolic handling authentication and billing across the catalog. The Flux model catalog includes:

  • GLM-5.3-Flash (hosted by Hyperbolic)

  • GLM-5.3

  • MiMo-V2.6-Flash

  • Kimi K3

  • Nemotron 3 Ultra

  • DeepSeek V4 Flash 0731

  • MiniMax M3

  • HY4 Preview

  • DeepSeek V4.1 Flash

  • Step 5 Preview

Call a model with the OpenAI SDK

The API supports Chat Completions with one completion per request and streaming over server-sent events. It is intended for applications, agent frameworks, and evaluation tools that accept a custom OpenAI-compatible endpoint. Reasoning, sampling, and output settings vary by model. Check the supported parameters for your selected model and allow enough output tokens for reasoning as well as the final response.

Inference requires a Pro account or an Organization, with prepaid credits. Requests are declined once the balance runs out, and API keys created for an organization draw on the organization’s balance. Create or reuse a Hyperbolic API key, install the OpenAI Python SDK, and set HYPERBOLIC_API_KEY in your environment. Use https://api.hyperbolic.ai/inference/v1 as the client base URL and glm-5.3-flash as the model ID.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.hyperbolic.ai/inference/v1",
    api_key=os.environ["HYPERBOLIC_API_KEY"],
)

stream = client.chat.completions.create(
    model="glm-5.3-flash",
    messages=[{"role": "user", "content": "Explain KV caching."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")

The API returns tokens as they are generated, so your application can display the response incrementally.

If you used Hyperbolic’s earlier inference API

To move from Hyperbolic’s earlier inference API, update your client’s base URL to https://api.hyperbolic.ai/inference/v1 and select a supported model ID. Use an API key that starts with sk_live_. Keys in the older format are not accepted, so create a new API key if needed. Then test the request parameters and streaming behavior your application relies on.

Check model pricing and usage

Inference usage is billed through your Hyperbolic account at the configured model rates, including separate cached-input pricing where supported. Hosted and externally routed models share Hyperbolic’s authentication and billing path. Rates vary by model, with input, cached-input, and output pricing where supported. Reasoning tokens are billed as output tokens. Review the published rate for your selected model before sending requests.

Request limits

Request-rate limits control how often your application sends calls. Model context windows and maximum output tokens determine how much each request can process and generate. Check the limits for your selected model and size requests to account for both reasoning and response tokens.

What’s next for Flux

We’re continuing to expand Flux with more open models and additional controls for production workloads. Planned improvements include:

  • More open models, informed by customer demand.

  • Per-account quotas and token-budget controls at the edge.

  • Automatic failover across model providers.

Forge integrates and virtualizes compute across providers, orchestrates workloads across clouds and clusters, and dynamically allocates GPU capacity through one unified control plane. Flux adds model serving with explicit security boundaries, token metering and asynchronous billing, and observability across the request path. Builders access it through an OpenAI-compatible API, with the infrastructure behind each response operated as part of the service.

Get started with Flux

Create or reuse an API key, configure your OpenAI client with https://api.hyperbolic.ai/inference/v1 and a supported model, and send your first request. Use the quickstart above to test streaming with your application.