AI inference & gateways on Azure
Model-serving runtimes, inference gateways and provider routing.
3 documented options on Microsoft Azure. Provider coverage reflects reviewed evidence, not an exhaustive compatibility claim.
Read this page as JSON or Markdown.
| Listing | Documented clouds | Deployment and infrastructure | Licensing | Installation | Operations and limitations |
|---|---|---|---|---|---|
| LiteLLM Proxy Self-Hosted | AWS, Azure, Google Cloud, Railway, Render | Customer-operated Servers / containers | Open Source / Open Core MIT outside the enterprise directory; enterprise code and features have separate license terms. | Official container image, Docker Compose or Kubernetes deployment; production infrastructure examples are documented. | Customer manages gateway availability, credentials, database, access policies, logs and upgrades, including pinning versions and verifying release artifacts with the documented version-appropriate Cosign method; model providers remain separate services.Scope and evidenceThe gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.
Cloud evidence reviewed 2026-11-12 |
| NVIDIA AI Enterprise | AWS, Azure, Google Cloud, Oracle Cloud, Alibaba Cloud, Tencent Cloud, Volcano Engine | Customer-operated Servers / containers | Commercial Only Commercial NVIDIA AI Enterprise subscription, consumption or perpetual license; most customer-cloud paths require BYOL. | NVIDIA AI Enterprise or GPU-optimized cloud VM images, standard instances with BYOL, or managed Kubernetes on supported cloud providers | Customer manages cloud GPU instances or Kubernetes, drivers, GPU Operator and container runtime, networking, storage, identities, NGC access, model services, monitoring, upgrades and license activation.Scope and evidenceThis entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.
Cloud evidence reviewed 2026-11-16 |
| vLLM Self-Hosted | AWS, Azure, Google Cloud | Customer-operated Servers / containers | Open Source / Open Core Apache 2.0 for vLLM; model weights and supporting dependencies have independent licenses and access conditions. | Official vLLM Docker image or supported Python installation; choose a compatible hardware/backend configuration. | Customer manages model acquisition, hardware capacity, endpoint protection, availability, upgrades and model/runtime compatibility.Scope and evidenceThis entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.
Cloud evidence reviewed 2026-11-12 |
Deployment evidence and category fit
LiteLLM Proxy Self-Hosted
Self-hosted model API gateway for routing, virtual keys, spend tracking and policy enforcement. It routes inference requests; it is not an inference engine for all supported models.
The gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.
- LiteLLM production deployment ↗
Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.
Official production guidance covers EKS, AKS and GKE with Helm, plus Terraform on AWS and Google Cloud. The proxy, database and Redis deployment is separate from the model services it calls.
Cloud evidence reviewed 2026-11-12. Full listing and qualifications
NVIDIA AI Enterprise
Includes supported inference and model-serving components for deploying generative-AI and machine-learning workloads.
This entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.
- NVIDIA AI Enterprise on AWS, Azure and Google Cloud ↗
Vendor / upstream documentation · Reviewed 2026-11-16. Vendor support commitment: not assessed; confirm the selected release and contract.
NVIDIA documents standard instances, NVIDIA VM images, GPU-optimized VM images, managed Kubernetes and marketplace OpenShift paths across AWS, Azure and Google Cloud. License inclusion varies by image and offer; standard instances and managed Kubernetes generally require BYOL.
Cloud evidence reviewed 2026-11-16. Full listing and qualifications
vLLM Self-Hosted
Model inference engine and OpenAI-compatible serving API for compatible model weights. Distinct from an API gateway that forwards calls to proprietary hosted models.
This entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.
- vLLM Production Stack deployment on Azure Kubernetes Service ↗
Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.
The project publishes Azure Kubernetes Service provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.
Cloud evidence reviewed 2026-11-12. Full listing and qualifications
Deployment and procurement checks
- Private offers
- Microsoft Marketplace private offers negotiate product pricing and terms. Confirm the exact offer, billing scope and benefit eligibility rather than assuming all purchases count toward a commitment. Provider source ↗ · Reviewed 2026-11-14
- Private connectivity
- Azure Private Link uses private endpoints in a virtual network for supported Azure and partner services. Check application support, DNS and service availability. Provider source ↗ · Reviewed 2026-11-14
- Cross-tenant operations
- Azure Lighthouse supports delegated management with customer-controlled scopes and actions. Agree the roles, audit requirements and revocation plan for the selected vendor. Provider source ↗ · Reviewed 2026-11-14
- Government environments
- Azure Government uses isolated US datacenters and networks. Commercial Azure documentation alone does not establish a product path or compliance in Government. Provider source ↗ · Reviewed 2026-11-14
For every provider, confirm marketplace/private-offer terms, commitment eligibility, regional service availability, private connectivity and any vendor access against your selected deployment. Provider capabilities do not automatically establish application support.
Plan this deployment
Account boundary: Subscription and resource group. Access: Microsoft Entra identities and Azure RBAC. Network: Virtual Network (VNet).
Provider documentation ↗ · Guidance reviewed 2026-11-10. Confirm the application edition, region, support access, backup locations and operating responsibilities before choosing.
Compare this category across clouds · Search these listings · Compare 2–4 listings