tenantplaneBook a demo Tune the serving layer around your workload.
Inference optimization is a separately scoped engagement: profile one model and request mix, compare serving configurations, and hand over measurements with the selected settings. Review precision, memory use, batching, replicas, and tensor parallelism against agreed latency and task-quality limits.
You receive the test configuration, baseline and comparison results, and a deployment recommendation. Scope and price follow a workload review; a speedup is measured, not assumed.
What the serving stack makes possible.
Published GPU results from the teams building the infrastructure. Different workloads answer different questions: per-user speed, aggregate capacity, and the value of a better serving topology.
25,000
total tokens / second / GPUHigh-throughput serving
vLLM reports 25K total TPS/GPU for Qwen3.5 on a disaggregated GB200 cluster.
- Model
- Qwen3.5-397B-A17B-NVFP4
- Hardware
- GB200 cluster with NVLink72; 16–24 GPUs across the configuration sweep
- Workload
- 8,192 input / 1,024 output tokens; random dataset, random_range_ratio=0.8; concurrency 64–5,120; prefix caching disabled
Configuration & measurement notes
Runtime: vLLM nightly-d223c90; Dynamo 1.2.0.dev20260526; srt-slurm v1.0.32
Precision: NVFP4 model weights
Comparison: Absolute peak across the reported configuration and concurrency sweep; no cross-runtime comparison
Total TPS counts prompt and generated tokens. One 8-GPU decode endpoint was paired with 4–8 two-GPU prefill endpoints. This is a peak from the sweep, not a single-GPU deployment. Output streaming used 100-token chunks; low-concurrency latency was not measured.
>250
tokens / second / userLow-concurrency generation
NVIDIA reports over 250 tokens/s/user for DeepSeek-R1 on one eight-GPU DGX B200.
- Model
- DeepSeek-R1 671B
- Hardware
- 1 DGX B200 system; 8 NVIDIA B200 GPUs
- Workload
- Concurrency 1; 1,024 input / 2,048 output tokens
Configuration & measurement notes
Runtime: Internal TensorRT-LLM build, March 2025 measurement
Precision: FP4
Comparison: Absolute reported per-user generation rate; not the maximum-throughput operating point
The source reports this per-user rate separately from its maximum-throughput result. The build was internal and the measurement is dated March 2025. It does not establish application response time or throughput at higher concurrency.
4.2×
chat throughput vs. tensor parallelTopology changes the result
Microsoft reports 4.2× chat throughput by giving each H100 its own Llama 3.1 8B model replica.
- Model
- Llama 3.1 8B Instruct
- Hardware
- 1 Azure ND-H100-v5 node; 8 NVIDIA H100 GPUs
- Workload
- Chat profile from share_gpt_turns.json; output min=50, max=800, variance=100; chunked prefill and prefix caching enabled
Configuration & measurement notes
Runtime: vLLM; Hugging Face inference benchmarker
Precision: Article evaluates FP16 and FP8; precision for this ratio is not separately specified in the conclusion
Comparison: Eight single-GPU model replicas (TP=1) versus one model distributed across all eight GPUs using tensor parallelism
The source reports 4.2× for chat, 5.2× for classification, and 1.9× for code generation. This example applies to a model that fits in one H100. It is not a general speedup guarantee or a result for large models requiring multiple GPUs. The exact request-rate point for the ratio is not stated in the article conclusion.
>30,000
tokens / second / systemMaximum-throughput operating point
NVIDIA reports over 30K tokens/s maximum throughput for DeepSeek-R1 on an eight-GPU DGX B200.
- Model
- DeepSeek-R1 671B
- Hardware
- 1 DGX B200 system; 8 NVIDIA B200 GPUs
- Workload
- 1,024 input / 2,048 output tokens; concurrency MAX (numeric concurrency not disclosed)
Configuration & measurement notes
Runtime: Internal TensorRT-LLM build, March 2025 measurement
Precision: FP4
Comparison: Absolute reported maximum system throughput
This is the high-concurrency operating point, separate from the >250 tokens/s/user result. The source does not disclose the numeric concurrency or an application latency SLO for this figure. Do not divide or combine the two results to infer concurrent users.
Your workload sets the acceptance criteria.
Peak throughput is useful for sizing a system. Your deployment also needs a response-time budget, representative inputs, and a defined operating envelope.
- Latency: time to first token and generation speed at the expected load.
- Capacity: concurrency, token lengths, memory use, and rejected requests.
- Application behavior: authentication, timeouts, invalid input, and service failure.
- Reproducibility: exact model, runtime, hardware, and configuration with the test record.
Download the acceptance checklist →
A repeatable release, not a one-off installation.
The handover ties the deployment files to the workload tests and recovery procedure. When a model or runtime changes, the same checks provide a baseline for the next release.