AI inference & gateways on Google Cloud

Model-serving runtimes, inference gateways and provider routing.

4 documented options on Google Cloud. Provider coverage reflects reviewed evidence, not an exhaustive compatibility claim.

ListingDocumented cloudsDeployment and infrastructureLicensingInstallationOperations and limitations
LiteLLM Proxy Self-HostedAWS, Azure, Google Cloud, Railway, RenderCustomer-operated
Servers / containers
Open Source / Open Core

MIT outside the enterprise directory; enterprise code and features have separate license terms.

Official container image, Docker Compose or Kubernetes deployment; production infrastructure examples are documented.Customer manages gateway availability, credentials, database, access policies, logs and upgrades, including pinning versions and verifying release artifacts with the documented version-appropriate Cosign method; model providers remain separate services.
Scope and evidence

The gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.

  • LiteLLM production deployment — Official production guidance covers EKS, AKS and GKE with Helm, plus Terraform on AWS and Google Cloud. The proxy, database and Redis deployment is separate from the model services it calls.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

  • LiteLLM Railway and Render deployment options — The current upstream deployment guide supplies Railway and Render deployment buttons in addition to its EKS/AKS/GKE instructions, followed by readiness and Admin UI verification. These are customer application deployments on managed platforms; configure persistent data, secrets and external model-provider endpoints separately.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

Cloud evidence reviewed 2026-11-12

NVIDIA AI EnterpriseAWS, Azure, Google Cloud, Oracle Cloud, Alibaba Cloud, Tencent Cloud, Volcano EngineCustomer-operated
Servers / containers
Commercial Only

Commercial NVIDIA AI Enterprise subscription, consumption or perpetual license; most customer-cloud paths require BYOL.

NVIDIA AI Enterprise or GPU-optimized cloud VM images, standard instances with BYOL, or managed Kubernetes on supported cloud providersCustomer manages cloud GPU instances or Kubernetes, drivers, GPU Operator and container runtime, networking, storage, identities, NGC access, model services, monitoring, upgrades and license activation.
Scope and evidence

This entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.

  • NVIDIA AI Enterprise on AWS, Azure and Google Cloud — NVIDIA documents standard instances, NVIDIA VM images, GPU-optimized VM images, managed Kubernetes and marketplace OpenShift paths across AWS, Azure and Google Cloud. License inclusion varies by image and offer; standard instances and managed Kubernetes generally require BYOL.

    Vendor / upstream documentation · Reviewed 2026-11-16. Vendor support commitment: not assessed; confirm the selected release and contract.

  • NVIDIA AI Enterprise on Oracle Cloud and Alibaba Cloud — NVIDIA's current cloud matrix includes standard instances on Oracle Cloud Infrastructure and Alibaba Cloud, NVIDIA AI Enterprise VM images on OCI and GPU-optimized VM images on Alibaba Cloud. Customers must confirm the supported component set, certified instance and license path.

    Vendor / upstream documentation · Reviewed 2026-11-16. Vendor support commitment: not assessed; confirm the selected release and contract.

  • NVIDIA AI Enterprise on Tencent Cloud standard instances — NVIDIA documents installing its licensed stack on customer standard instances using a certified Ubuntu operating system, supported NVIDIA drivers, Docker and Container Toolkit, then pulling NGC containers. This is a BYOL installation rather than an included-license marketplace image. Select an instance from the supported-cloud-services matrix; the guide states one license per cloud instance, which must be reconciled with the general licensing guide and the customer agreement.

    Vendor / upstream documentation · Reviewed 2026-11-16 · Scope: NVIDIA AI Enterprise standard-instance BYOL. Vendor support commitment: not assessed; confirm the selected release and contract.

  • NVIDIA AI Enterprise on ByteDance Volcano Engine standard instances — NVIDIA documents installing its licensed stack on customer standard instances using a certified Ubuntu operating system, supported NVIDIA drivers, Docker and Container Toolkit, then pulling NGC containers. This is a BYOL installation rather than an included-license marketplace image. Select an instance from the supported-cloud-services matrix; the guide states one license per cloud instance, which must be reconciled with the general licensing guide and the customer agreement.

    Vendor / upstream documentation · Reviewed 2026-11-16 · Scope: NVIDIA AI Enterprise standard-instance BYOL. Vendor support commitment: not assessed; confirm the selected release and contract.

Cloud evidence reviewed 2026-11-16

SGLangGoogle CloudCustomer-operated
Servers / containers
Open Source / Open Core

Apache License 2.0 for upstream SGLang. Model weights and hardware/runtime dependencies have their own licenses.

Install the package or use official containers; the reviewed primary installation path targets Linux with supported NVIDIA GPUs and compatible drivers.Customer manages accelerator capacity, model artifacts, runtime compatibility, endpoint access, including isolation of debug and dumper routes, monitoring, and upgrades.
Scope and evidence

Model downloads may require external registries and gated-model credentials. Pin compatible model/runtime/image versions and review model-specific permissions. Generic container instructions do not establish deployment badges for every hyperscaler. CVE-2026-111391 rates SGLang 0.5.10 through 0.5.22 critical (CVSS v4.0 9.2) for Python code execution through the debug dumper filter. The upstream report demonstrates the shared-API path with DUMPER_SERVER_PORT=reuse: an ordinary API-key holder can reach it when no separate admin API key is configured; access can be anonymous when no API key is configured. With an admin API key configured, that key is required. Exploitation also requires access to the dumper configuration and a subsequent model execution that reaches a dumper hook. Upstream issue #44188 remains open, and the v0.5.23 tag still contains the eval-based filter path even though that tag is outside the CVE record's enumerated affected range. Do not enable or expose the debug dumper endpoint; restrict it at the network layer and verify an upstream fix before treating a later version as remediated.

  • SGLang compatibility with GKE Inference Gateway — Google explicitly supports SGLang as an alternative model server in its customer-operated multi-cluster GKE inference architecture. The worked manifests use vLLM; SGLang requires its own deployment, serving port, metric mapping and compatible Endpoint Picker image. Review GPU/model requirements, licenses and resource availability.

    Provider guide · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

Cloud evidence reviewed 2026-11-12

vLLM Self-HostedAWS, Azure, Google CloudCustomer-operated
Servers / containers
Open Source / Open Core

Apache 2.0 for vLLM; model weights and supporting dependencies have independent licenses and access conditions.

Official vLLM Docker image or supported Python installation; choose a compatible hardware/backend configuration.Customer manages model acquisition, hardware capacity, endpoint protection, availability, upgrades and model/runtime compatibility.
Scope and evidence

This entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.

  • vLLM Production Stack deployment on Amazon EKS — The project publishes Amazon EKS provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

  • vLLM Production Stack deployment on Azure Kubernetes Service — The project publishes Azure Kubernetes Service provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

  • vLLM Production Stack deployment on Google Kubernetes Engine — The project publishes Google Kubernetes Engine provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

Cloud evidence reviewed 2026-11-12

Deployment evidence and category fit

LiteLLM Proxy Self-Hosted

Self-hosted model API gateway for routing, virtual keys, spend tracking and policy enforcement. It routes inference requests; it is not an inference engine for all supported models.

The gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.

  • LiteLLM production deployment ↗

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

    Official production guidance covers EKS, AKS and GKE with Helm, plus Terraform on AWS and Google Cloud. The proxy, database and Redis deployment is separate from the model services it calls.

Cloud evidence reviewed 2026-11-12. Full listing and qualifications

NVIDIA AI Enterprise

Includes supported inference and model-serving components for deploying generative-AI and machine-learning workloads.

This entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.

  • NVIDIA AI Enterprise on AWS, Azure and Google Cloud ↗

    Vendor / upstream documentation · Reviewed 2026-11-16. Vendor support commitment: not assessed; confirm the selected release and contract.

    NVIDIA documents standard instances, NVIDIA VM images, GPU-optimized VM images, managed Kubernetes and marketplace OpenShift paths across AWS, Azure and Google Cloud. License inclusion varies by image and offer; standard instances and managed Kubernetes generally require BYOL.

Cloud evidence reviewed 2026-11-16. Full listing and qualifications

SGLang

Model-serving framework for LLM and multimodal inference; an inference runtime rather than a full multi-provider governance gateway.

Model downloads may require external registries and gated-model credentials. Pin compatible model/runtime/image versions and review model-specific permissions. Generic container instructions do not establish deployment badges for every hyperscaler. CVE-2026-111391 rates SGLang 0.5.10 through 0.5.22 critical (CVSS v4.0 9.2) for Python code execution through the debug dumper filter. The upstream report demonstrates the shared-API path with DUMPER_SERVER_PORT=reuse: an ordinary API-key holder can reach it when no separate admin API key is configured; access can be anonymous when no API key is configured. With an admin API key configured, that key is required. Exploitation also requires access to the dumper configuration and a subsequent model execution that reaches a dumper hook. Upstream issue #44188 remains open, and the v0.5.23 tag still contains the eval-based filter path even though that tag is outside the CVE record's enumerated affected range. Do not enable or expose the debug dumper endpoint; restrict it at the network layer and verify an upstream fix before treating a later version as remediated.

  • SGLang compatibility with GKE Inference Gateway ↗

    Provider guide · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

    Google explicitly supports SGLang as an alternative model server in its customer-operated multi-cluster GKE inference architecture. The worked manifests use vLLM; SGLang requires its own deployment, serving port, metric mapping and compatible Endpoint Picker image. Review GPU/model requirements, licenses and resource availability.

Cloud evidence reviewed 2026-11-12. Full listing and qualifications

vLLM Self-Hosted

Model inference engine and OpenAI-compatible serving API for compatible model weights. Distinct from an API gateway that forwards calls to proprietary hosted models.

This entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.

  • vLLM Production Stack deployment on Google Kubernetes Engine ↗

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

    The project publishes Google Kubernetes Engine provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.

Cloud evidence reviewed 2026-11-12. Full listing and qualifications

Deployment and procurement checks

Private offers
Cloud Marketplace vendors can negotiate a private offer for a specified Cloud Billing account. Check duration, renewal terms and commitment treatment in the actual agreement. Provider source ↗ · Reviewed 2026-11-14
Private connectivity
Private Service Connect connects VPC consumers to supported managed or producer services. Confirm the actual product endpoint and network/DNS architecture. Provider source ↗ · Reviewed 2026-11-14
External workload access
Workload Identity Federation can grant external workloads scoped Google Cloud access without long-lived service-account keys. Validate the trust configuration and exact roles. Provider source ↗ · Reviewed 2026-11-14
Regulated environments
Assured Workloads applies control packages for residency and access boundaries. Confirm supported services, regions and the product deployment; it is not automatic product certification. Provider source ↗ · Reviewed 2026-11-14

For every provider, confirm marketplace/private-offer terms, commitment eligibility, regional service availability, private connectivity and any vendor access against your selected deployment. Provider capabilities do not automatically establish application support.

Plan this deployment

Account boundary: Google Cloud project. Access: IAM policies and service accounts. Network: VPC network.

Provider documentation ↗ · Guidance reviewed 2026-11-10. Confirm the application edition, region, support access, backup locations and operating responsibilities before choosing.

Compare this category across clouds · Search these listings · Compare 2–4 listings