vLLM Self-Hosted
Self-hosted model inference and serving engine with OpenAI-compatible APIs.
5 sources checked by an editor on · Product website ↗
Read this page as JSON or Markdown.
Deployment terms: BYOC · control plane · data plane · private offer
· Open requirements shortlist · Estimate operating costs
Compare this listing with another →
What lands where
Supplied by
Vendor or project
Official vLLM Docker image or supported Python installation; choose a compatible hardware/backend configuration.
Runs in
Documented customer environment
Customer-operated model-serving process, model weights and inference hardware
- AWS: Customer cloud account
- Azure: Customer cloud account
- Google Cloud: Customer cloud account
- AWS
- Azure
- Google Cloud
Servers / containers
Operated by
Your team
Customer manages model acquisition, hardware capacity, endpoint protection, availability, upgrades and model/runtime compatibility.
Deployment boundaries by provider
- AWS: Customer cloud account · Customer-operated
Resources in the customer's cloud account, subscription, project or tenancy. Managed cloud services still have shared responsibilities.
vLLM Production Stack deployment on Amazon EKS ↗ · Source checked 2026-11-12
- Azure: Customer cloud account · Customer-operated
Resources in the customer's cloud account, subscription, project or tenancy. Managed cloud services still have shared responsibilities.
vLLM Production Stack deployment on Azure Kubernetes Service ↗ · Source checked 2026-11-12
- Google Cloud: Customer cloud account · Customer-operated
Resources in the customer's cloud account, subscription, project or tenancy. Managed cloud services still have shared responsibilities.
vLLM Production Stack deployment on Google Kubernetes Engine ↗ · Source checked 2026-11-12
Application operations and account ownership are separate from infrastructure operations. Confirm external services, telemetry, support access and backup destinations for the chosen path.
| Install method | Official vLLM Docker image or supported Python installation; choose a compatible hardware/backend configuration. |
|---|---|
| Infrastructure | Servers / containers vLLM is a running inference-server workload. Hardware support and capacity depend on the model and selected backend; the documented container paths include accelerator-specific images, and CPU support is separately documented. In v0.34.0, the KV-cache allocator sizes from free memory after warmup and defaults to gpu_memory_utilization=1.0; revalidate GPU headroom, and set 0.9 or disable extensible cache to retain prior sizing. ROCm 10.0 replaces 7.2 for published images and wheels. |
| Licensing model | Open Source / Open Core The reviewed inference runtime is Apache-2.0 licensed. Its license does not grant rights to the model weights served by that runtime; check model and dependency terms separately. |
| License | Apache 2.0 for vLLM; model weights and supporting dependencies have independent licenses and access conditions. |
Buyer diligence
Answers apply to the reviewed edition and deployment. Expand a row for qualifications and sources.
How to read these facts
Not researched means no field-specific review is recorded. Not confirmed means sources were checked but did not establish an answer. Availability, eligibility and edition-dependent findings retain the scope given in their evidence. A vendor’s hosted-service certification does not automatically cover your deployment. Evidence coverage is not a feature score.
5 of 12 fields researched (42%) · 3 documented · 2 unresolved. Evidence coverage is not a feature score.
- Single sign-on
Not confirmedEvidence and scope for Single sign-on
The security guide documents API-key protection for specified HTTP path prefixes and warns that other sensitive endpoints remain unprotected. It does not establish a native SAML/OIDC user sign-on workflow. Use a reviewed gateway/network design; an API bearer key alone is not corporate SSO or complete endpoint protection.
Upstream inference-server auth surface; federated user integration unresolved · Source ↗ · Reviewed 2026-11-20
Sources checked — unresolved · Editor reviewed.
- Deployment artifact / IaC
DocumentedEvidence and scope for Deployment artifact / IaC
The stable deployment guide links the official vllm/vllm-openai image for a CUDA-backed OpenAI-compatible server. Select a release/architecture tag and compatible GPU drivers; model memory, credentials and optional dependencies need separate review.
Stable upstream CUDA image path; not the latest developer-preview guide or a GPU sizing guarantee · Source ↗ · Deployment artifact ↗ (select a stable version and architecture tag; latest is moving) · Reviewed 2026-11-14
Documented · Editor reviewed.
- Minimum footprint / sizing
Not confirmedEvidence and scope for Minimum footprint / sizing
The stable memory guide sizes serving around model weights, precision/quantization, tensor parallelism, context length, batch size and cache. A supported accelerator alone does not establish sufficient memory. Select the exact model and target throughput/latency, then benchmark VRAM/RAM and HA replica requirements.
Model-specific inference sizing; no universal numeric footprint established · Source ↗ · Reviewed 2026-11-20
Sources checked — unresolved · Editor reviewed.
- License pricing
DocumentedEvidence and scope for License pricing
The reviewed upstream core is Apache-2.0 licensed. Optional services, bundled dependencies, commercial distributions and support can have separate terms. This records the source licence, not an infrastructure or operating-cost estimate. Model weights and external model APIs require their own review.
Upstream core licence; commercial features, support and hosting priced separately · Source ↗ · Reviewed 2026-11-20
Documented · Editor reviewed.
- Release snapshot / cadence
DocumentedEvidence and scope for Release snapshot / cadence
The reviewed upstream release is v0.34.0, published 2026-11-16. This is a dated release snapshot, not a support designation, a future cadence promise or approval to replace a provider package.
Upstream vllm-project/vllm release record; commercial editions, LTS branches and provider packages can differ · Source ↗ · Reviewed 2026-11-17
Documented · Editor reviewed.
Show all fields (7 not researched)
- Marketplace delivery
- Not researched Supply a public source.
- Cloud commitment eligibility
- Not researched Supply a public source.
- SCIM provisioning
- Not researched Supply a public source.
- Compliance attestations
- Not researched Supply a public source.
- Security controls
- Not researched Supply a public source.
- Air-gapped operation
- Not researched Supply a public source.
- Operating cost estimate
- Not researched Supply a public source.
Scope & considerations
This entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.
Category fit
- AI inference & gateways
- Model inference engine and OpenAI-compatible serving API for compatible model weights. Distinct from an API gateway that forwards calls to proprietary hosted models.
- AI tools & infrastructure
- Self-hosted model inference and serving engine with OpenAI-compatible APIs.
Compare within AI inference & gateways
- LiteLLM Proxy Self-HostedSelf-hosted AI API gateway providing model routing, access controls and usage tracking.
- AWS
- Azure
- Google Cloud
- Railway
- Render
- LocalAILocal model inference service exposing configurable AI capabilities through compatible API endpoints.
- AWS
- NVIDIA AI EnterpriseCommercial enterprise AI software suite deployed on customer cloud GPU instances or managed Kubernetes clusters.
- AWS
- Azure
- Google Cloud
- Oracle Cloud
- Alibaba Cloud
- Tencent Cloud
- Volcano Engine
- SGLangModel-serving framework for running supported language and multimodal models on customer-controlled compute.
- Google Cloud
These products address related workflows; read their category-fit notes before comparing scope. See all of AI inference & gateways →
Installation outline
- Choose the documented deployment option and a dedicated cloud environment. Follow the launch methodology.
- Review the vendor's installation documentation, required permissions and outbound data flows.
- Deploy to a non-production environment; configure identity, networking and monitoring.
- Test backup recovery and upgrades before promoting to production.
Evidence trail
Every claim on this page traces to the sources below. They document the deployment option; they do not constitute a security certification.
- Deployment sourcesChecked
- Official Docker serving images ↗docs.vllm.ai
- Online serving ↗docs.vllm.ai
- Usage statistics and opt-out ↗docs.vllm.ai
- Apache 2.0 license ↗github.com
- vLLM v0.34.0 breaking upgrade notes ↗github.com
- Cloud deployment evidenceChecked
The linked sources describe each deployment path, edition and support scope. Community templates and partner statements are identified in the notes.
- vLLM Production Stack deployment on Amazon EKS ↗ · AWS
Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.
The project publishes Amazon EKS provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.
- vLLM Production Stack deployment on Azure Kubernetes Service ↗ · Azure
Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.
The project publishes Azure Kubernetes Service provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.
- vLLM Production Stack deployment on Google Kubernetes Engine ↗ · Google Cloud
Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.
The project publishes Google Kubernetes Engine provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.
- vLLM Production Stack deployment on Amazon EKS ↗ · AWS
- Infrastructure evidenceChecked
- Licensing evidenceChecked