AI inference & gateways on AWS

Model-serving runtimes, inference gateways and provider routing.

4 documented options on Amazon Web Services. Provider coverage reflects reviewed evidence, not an exhaustive compatibility claim.

ListingDocumented cloudsDeployment and infrastructureLicensingInstallationOperations and limitations
LiteLLM Proxy Self-HostedAWS, Azure, Google Cloud, Railway, RenderCustomer-operated
Servers / containers
Open Source / Open Core

MIT outside the enterprise directory; enterprise code and features have separate license terms.

Official container image, Docker Compose or Kubernetes deployment; production infrastructure examples are documented.Customer manages gateway availability, credentials, database, access policies, logs and upgrades, including pinning versions and verifying release artifacts with the documented version-appropriate Cosign method; model providers remain separate services.
Scope and evidence

The gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.

  • LiteLLM production deployment — Official production guidance covers EKS, AKS and GKE with Helm, plus Terraform on AWS and Google Cloud. The proxy, database and Redis deployment is separate from the model services it calls.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

  • LiteLLM Railway and Render deployment options — The current upstream deployment guide supplies Railway and Render deployment buttons in addition to its EKS/AKS/GKE instructions, followed by readiness and Admin UI verification. These are customer application deployments on managed platforms; configure persistent data, secrets and external model-provider endpoints separately.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

Cloud evidence reviewed 2026-11-12

LocalAIAWSCustomer-operated
Servers / containers
Open Source / Open Core

MIT License for upstream LocalAI. Model artifacts and inference backends can have different license and hardware requirements.

Run official CPU/GPU Docker images or the documented binaries and configure models, storage and endpoint access.Customer manages models, backends, hardware capacity, API authentication, persistent storage and upgrades.
Scope and evidence

CPU availability does not guarantee acceptable performance for every model. Model galleries and artifact downloads contact external registries; optional integrations and remote backends can change data boundaries.

  • LocalAI and Flowise on AWS EKS with Pulumi — This Pulumi-authored deployment recipe, linked by the LocalAI project, runs LocalAI inference and Flowise on customer EKS infrastructure. The demonstrated configuration uses CPU inference; review model licenses, current images, resource sizing and network exposure rather than assuming GPU support or production hardening.

    Third party deployment guide · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

Cloud evidence reviewed 2026-11-12

NVIDIA AI EnterpriseAWS, Azure, Google Cloud, Oracle Cloud, Alibaba Cloud, Tencent Cloud, Volcano EngineCustomer-operated
Servers / containers
Commercial Only

Commercial NVIDIA AI Enterprise subscription, consumption or perpetual license; most customer-cloud paths require BYOL.

NVIDIA AI Enterprise or GPU-optimized cloud VM images, standard instances with BYOL, or managed Kubernetes on supported cloud providersCustomer manages cloud GPU instances or Kubernetes, drivers, GPU Operator and container runtime, networking, storage, identities, NGC access, model services, monitoring, upgrades and license activation.
Scope and evidence

This entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.

  • NVIDIA AI Enterprise on AWS, Azure and Google Cloud — NVIDIA documents standard instances, NVIDIA VM images, GPU-optimized VM images, managed Kubernetes and marketplace OpenShift paths across AWS, Azure and Google Cloud. License inclusion varies by image and offer; standard instances and managed Kubernetes generally require BYOL.

    Vendor / upstream documentation · Reviewed 2026-11-16. Vendor support commitment: not assessed; confirm the selected release and contract.

  • NVIDIA AI Enterprise on Oracle Cloud and Alibaba Cloud — NVIDIA's current cloud matrix includes standard instances on Oracle Cloud Infrastructure and Alibaba Cloud, NVIDIA AI Enterprise VM images on OCI and GPU-optimized VM images on Alibaba Cloud. Customers must confirm the supported component set, certified instance and license path.

    Vendor / upstream documentation · Reviewed 2026-11-16. Vendor support commitment: not assessed; confirm the selected release and contract.

  • NVIDIA AI Enterprise on Tencent Cloud standard instances — NVIDIA documents installing its licensed stack on customer standard instances using a certified Ubuntu operating system, supported NVIDIA drivers, Docker and Container Toolkit, then pulling NGC containers. This is a BYOL installation rather than an included-license marketplace image. Select an instance from the supported-cloud-services matrix; the guide states one license per cloud instance, which must be reconciled with the general licensing guide and the customer agreement.

    Vendor / upstream documentation · Reviewed 2026-11-16 · Scope: NVIDIA AI Enterprise standard-instance BYOL. Vendor support commitment: not assessed; confirm the selected release and contract.

  • NVIDIA AI Enterprise on ByteDance Volcano Engine standard instances — NVIDIA documents installing its licensed stack on customer standard instances using a certified Ubuntu operating system, supported NVIDIA drivers, Docker and Container Toolkit, then pulling NGC containers. This is a BYOL installation rather than an included-license marketplace image. Select an instance from the supported-cloud-services matrix; the guide states one license per cloud instance, which must be reconciled with the general licensing guide and the customer agreement.

    Vendor / upstream documentation · Reviewed 2026-11-16 · Scope: NVIDIA AI Enterprise standard-instance BYOL. Vendor support commitment: not assessed; confirm the selected release and contract.

Cloud evidence reviewed 2026-11-16

vLLM Self-HostedAWS, Azure, Google CloudCustomer-operated
Servers / containers
Open Source / Open Core

Apache 2.0 for vLLM; model weights and supporting dependencies have independent licenses and access conditions.

Official vLLM Docker image or supported Python installation; choose a compatible hardware/backend configuration.Customer manages model acquisition, hardware capacity, endpoint protection, availability, upgrades and model/runtime compatibility.
Scope and evidence

This entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.

  • vLLM Production Stack deployment on Amazon EKS — The project publishes Amazon EKS provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

  • vLLM Production Stack deployment on Azure Kubernetes Service — The project publishes Azure Kubernetes Service provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

  • vLLM Production Stack deployment on Google Kubernetes Engine — The project publishes Google Kubernetes Engine provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

Cloud evidence reviewed 2026-11-12

Deployment evidence and category fit

LiteLLM Proxy Self-Hosted

Self-hosted model API gateway for routing, virtual keys, spend tracking and policy enforcement. It routes inference requests; it is not an inference engine for all supported models.

The gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.

  • LiteLLM production deployment ↗

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

    Official production guidance covers EKS, AKS and GKE with Helm, plus Terraform on AWS and Google Cloud. The proxy, database and Redis deployment is separate from the model services it calls.

Cloud evidence reviewed 2026-11-12. Full listing and qualifications

LocalAI

Local inference service with OpenAI-compatible API endpoints and configurable model backends; compatible API support does not require a hosted OpenAI service.

CPU availability does not guarantee acceptable performance for every model. Model galleries and artifact downloads contact external registries; optional integrations and remote backends can change data boundaries.

  • LocalAI and Flowise on AWS EKS with Pulumi ↗

    Third party deployment guide · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

    This Pulumi-authored deployment recipe, linked by the LocalAI project, runs LocalAI inference and Flowise on customer EKS infrastructure. The demonstrated configuration uses CPU inference; review model licenses, current images, resource sizing and network exposure rather than assuming GPU support or production hardening.

Cloud evidence reviewed 2026-11-12. Full listing and qualifications

NVIDIA AI Enterprise

Includes supported inference and model-serving components for deploying generative-AI and machine-learning workloads.

This entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.

  • NVIDIA AI Enterprise on AWS, Azure and Google Cloud ↗

    Vendor / upstream documentation · Reviewed 2026-11-16. Vendor support commitment: not assessed; confirm the selected release and contract.

    NVIDIA documents standard instances, NVIDIA VM images, GPU-optimized VM images, managed Kubernetes and marketplace OpenShift paths across AWS, Azure and Google Cloud. License inclusion varies by image and offer; standard instances and managed Kubernetes generally require BYOL.

Cloud evidence reviewed 2026-11-16. Full listing and qualifications

vLLM Self-Hosted

Model inference engine and OpenAI-compatible serving API for compatible model weights. Distinct from an API gateway that forwards calls to proprietary hosted models.

This entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.

  • vLLM Production Stack deployment on Amazon EKS ↗

    Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.

    The project publishes Amazon EKS provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.

Cloud evidence reviewed 2026-11-12. Full listing and qualifications

Deployment and procurement checks

Private offers
AWS Marketplace private offers can contain negotiated pricing and terms for a buyer account. Confirm the product delivery model and your agreement; commitment credit is not established by a directory badge. Provider source ↗ · Reviewed 2026-11-14
Private connectivity
PrivateLink provides private endpoints to supported services and resources across VPC/account boundaries. Confirm that the selected product exposes a compatible endpoint service. Provider source ↗ · Reviewed 2026-11-14
Cross-account operations
Use an IAM role with scoped permissions and a third-party external ID where applicable. Review the trust principal, support access and revocation separately from installation access. Provider source ↗ · Reviewed 2026-11-14
Government environments
AWS GovCloud (US) is a distinct isolated-region environment. A commercial-region product deployment does not establish availability or certification there. Provider source ↗ · Reviewed 2026-11-14

For every provider, confirm marketplace/private-offer terms, commitment eligibility, regional service availability, private connectivity and any vendor access against your selected deployment. Provider capabilities do not automatically establish application support.

Plan this deployment

Account boundary: AWS account. Access: IAM roles and policies. Network: VPC.

Provider documentation ↗ · Guidance reviewed 2026-11-10. Confirm the application edition, region, support access, backup locations and operating responsibilities before choosing.

Compare this category across clouds · Search these listings · Compare 2–4 listings