Deploy a model

Name a model. Routing, health checks and metrics are set up for you.

Model

Replicas: 1 (more replicas, more requests at once)
GPUs per replica: 1 (the model is split across them)
Machines per replica: 1 (fits on one machine)
Team: …
Name
Engine
GPU line: Wait in the team's line for GPUs (off: starts now on any free GPUs, outside the team's quota)

Uses 1 GPU in total. You get an OpenAI-compatible endpoint behind llm-d's router.

{ } API call

curl -X POST https://runai.company.internal/cluster-api/api/v1/llmd/deployments \
  -H "Authorization: Bearer $RUNAI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"project":"","name":"","model":"","engine":"vllm","replicas":1,"gpusPerReplica":1}'

Get a token with POST /api/v1/token using your credentials or a service account.

What happens when you click

Checks you're allowed Run:ai
Confirms you can deploy to your team, records who deployed what, and creates the model servers as a Run:ai workload.
Skips the line Kueue · skipped
Starts right away on any free GPUs. It isn't counted against your team's quota, and Kueue won't preempt it.
Picks machines kube-scheduler
The standard Kubernetes scheduler puts each replica on a machine with 1 free GPU.
Loads the model vLLM
Downloads the weights and serves them on each replica.
Routes every request llm-d
llm-d's router (Envoy plus its Endpoint Picker) sends each request to the replica that already has the most of the prompt cached and the shortest queue.
You get an endpoint you
An OpenAI-compatible URL, live logs, latency and cache metrics. Scale replicas up or down any time.