tenantplaneBook a demo An on-premises LLM deployment is ready when the customer’s team can use it, control access, and recover it under the conditions they agreed to. Loading a model onto a GPU is one step in that process.
Use this checklist during discovery and acceptance planning. It is a planning framework; the evidence for a particular deployment must come from that environment and workload.
Define the workload and acceptance target
Write down the first business task the system will support. Identify the users, data sources, expected request rate, busiest period, typical input length, and maximum output length. Agree on representative examples and unacceptable failures.
| Area | Record before implementation | Evidence to request at acceptance |
|---|---|---|
| Task quality | Example inputs and evaluation criteria | Reviewed results on representative tasks |
| Capacity | Prompt lengths, concurrency, response target | Load results for the agreed workload |
| Permissions | User groups and document access rules | Allowed, denied, and revoked-access tests |
| Network | Approved inbound and outbound connections | Observed connections during normal and failure paths |
| Recovery | Acceptable outage and data-loss limits | Timed restart, restore, and rollback exercises |
Separate the model’s output quality from the system’s operating performance. A fast answer that quotes an inaccessible document fails acceptance even if the GPU is functioning correctly.
Confirm hardware and software compatibility
Check the exact accelerator, available memory, operating system, driver, runtime, serving engine, and model format together. Reserve capacity for the model, runtime overhead, and request state. Also account for embeddings, reranking, or speech models sharing the machine.
Size against the workload rather than parameter count alone. vLLM documents context length and maximum sequence count as memory-control variables. Reducing either changes what the service can handle and must be reflected in the acceptance target. vLLM memory guidance.
Confirm power, cooling, storage, network throughput, physical access, and replacement arrangements. Ask what happens when the only accelerator fails. A single-machine deployment may suit a pilot with a tolerable outage; it does not provide hardware redundancy.
Establish model provenance and update access
Record the source, revision, checksums, tokenizer, configuration, and license for every model artifact. Review the actual license and any redistribution or commercial-use conditions before packaging weights for a customer.
Check download access separately from licensing. Hugging Face’s documentation notes that gated-model access is granted to individual users, rather than entire organizations. A developer’s successful download does not prove the customer’s automation can retrieve the same artifacts. Hugging Face gated models.
Document how approved artifacts enter the environment, who approves them, how integrity is checked, and which previous release is retained for rollback. An offline environment needs an update process as much as a connected one.
Verify the network boundary
List every outbound destination: registries, model repositories, package updates, identity, time synchronization, certificate services, telemetry, and support tooling. Include components surrounding inference, such as document extraction and embeddings.
For an air-gapped requirement, test installation from approved local artifacts and operation without internet access. For connected environments, restrict outbound access to the documented requirements and inspect failure behavior.
If Kubernetes NetworkPolicy is part of the design, verify that the installed networking implementation enforces it. Creating a policy resource alone has no effect without a supporting controller. Kubernetes NetworkPolicy documentation.
Separate users, administrators, and tools
Map application users to approved identities. Assign separate service identities to components with different privileges. Describe how access is revoked, how support access is approved, and where credentials are stored and rotated.
Place model-serving and administrative interfaces behind the intended access controls. vLLM’s security documentation warns against relying exclusively on its API-key option because some endpoints have different protection. Review the precise engine and version being deployed. vLLM security guidance.
For retrieval, enforce document authorization before restricted content reaches the model. Test two users with different permissions, group changes, deleted documents, and direct API access. For tool use, restrict the executing service’s permissions and require approval for consequential actions where the workflow calls for it. Prompt instructions do not replace access controls.
Prove recovery and assign the handover
Exercise a service restart, unavailable model artifact, full storage volume, failed update, and lost dependency. Confirm which data needs backup and demonstrate a restore into an approved recovery environment. Record what state can be reconstructed and what would be lost.
Deliver configuration, artifact manifests, installation and rollback instructions, monitoring thresholds, an issue register, and acceptance results. Have the customer’s named operator follow the runbook. Assign responsibility for hardware, operating-system patches, model upgrades, backups, certificates, and incident response, including support outside business hours.
Use the private AI cost guide to budget those responsibilities. If the target is a cloud account, the customer VPC guide covers the hosting and access decisions.
See Tenantplane’s private AI deployment service or discuss your environment to discuss a scoped implementation and its acceptance checks.