AI inference & gateways on AWS
Model-serving runtimes, inference gateways and provider routing.
4 documented options on Amazon Web Services. Provider coverage reflects reviewed evidence, not an exhaustive compatibility claim.
Read this page as JSON or Markdown.
| Listing | Documented clouds | Deployment and infrastructure | Licensing | Installation | Operations and limitations |
|---|---|---|---|---|---|
| LiteLLM Proxy Self-Hosted | AWS, Azure, Google Cloud, Railway, Render | Customer-operated Servers / containers | Open Source / Open Core MIT outside the enterprise directory; enterprise code and features have separate license terms. | Official container image, Docker Compose or Kubernetes deployment; production infrastructure examples are documented. | Customer manages gateway availability, credentials, database, access policies, logs and upgrades, including pinning versions and verifying release artifacts with the documented version-appropriate Cosign method; model providers remain separate services.Scope and evidenceThe gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.
Cloud evidence reviewed 2026-11-12 |
| LocalAI | AWS | Customer-operated Servers / containers | Open Source / Open Core MIT License for upstream LocalAI. Model artifacts and inference backends can have different license and hardware requirements. | Run official CPU/GPU Docker images or the documented binaries and configure models, storage and endpoint access. | Customer manages models, backends, hardware capacity, API authentication, persistent storage and upgrades.Scope and evidenceCPU availability does not guarantee acceptable performance for every model. Model galleries and artifact downloads contact external registries; optional integrations and remote backends can change data boundaries.
Cloud evidence reviewed 2026-11-12 |
| NVIDIA AI Enterprise | AWS, Azure, Google Cloud, Oracle Cloud, Alibaba Cloud, Tencent Cloud, Volcano Engine | Customer-operated Servers / containers | Commercial Only Commercial NVIDIA AI Enterprise subscription, consumption or perpetual license; most customer-cloud paths require BYOL. | NVIDIA AI Enterprise or GPU-optimized cloud VM images, standard instances with BYOL, or managed Kubernetes on supported cloud providers | Customer manages cloud GPU instances or Kubernetes, drivers, GPU Operator and container runtime, networking, storage, identities, NGC access, model services, monitoring, upgrades and license activation.Scope and evidenceThis entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.
Cloud evidence reviewed 2026-11-16 |
| vLLM Self-Hosted | AWS, Azure, Google Cloud | Customer-operated Servers / containers | Open Source / Open Core Apache 2.0 for vLLM; model weights and supporting dependencies have independent licenses and access conditions. | Official vLLM Docker image or supported Python installation; choose a compatible hardware/backend configuration. | Customer manages model acquisition, hardware capacity, endpoint protection, availability, upgrades and model/runtime compatibility.Scope and evidenceThis entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.
Cloud evidence reviewed 2026-11-12 |
Deployment evidence and category fit
LiteLLM Proxy Self-Hosted
Self-hosted model API gateway for routing, virtual keys, spend tracking and policy enforcement. It routes inference requests; it is not an inference engine for all supported models.
The gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.
- LiteLLM production deployment ↗
Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.
Official production guidance covers EKS, AKS and GKE with Helm, plus Terraform on AWS and Google Cloud. The proxy, database and Redis deployment is separate from the model services it calls.
Cloud evidence reviewed 2026-11-12. Full listing and qualifications
LocalAI
Local inference service with OpenAI-compatible API endpoints and configurable model backends; compatible API support does not require a hosted OpenAI service.
CPU availability does not guarantee acceptable performance for every model. Model galleries and artifact downloads contact external registries; optional integrations and remote backends can change data boundaries.
- LocalAI and Flowise on AWS EKS with Pulumi ↗
Third party deployment guide · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.
This Pulumi-authored deployment recipe, linked by the LocalAI project, runs LocalAI inference and Flowise on customer EKS infrastructure. The demonstrated configuration uses CPU inference; review model licenses, current images, resource sizing and network exposure rather than assuming GPU support or production hardening.
Cloud evidence reviewed 2026-11-12. Full listing and qualifications
NVIDIA AI Enterprise
Includes supported inference and model-serving components for deploying generative-AI and machine-learning workloads.
This entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.
- NVIDIA AI Enterprise on AWS, Azure and Google Cloud ↗
Vendor / upstream documentation · Reviewed 2026-11-16. Vendor support commitment: not assessed; confirm the selected release and contract.
NVIDIA documents standard instances, NVIDIA VM images, GPU-optimized VM images, managed Kubernetes and marketplace OpenShift paths across AWS, Azure and Google Cloud. License inclusion varies by image and offer; standard instances and managed Kubernetes generally require BYOL.
Cloud evidence reviewed 2026-11-16. Full listing and qualifications
vLLM Self-Hosted
Model inference engine and OpenAI-compatible serving API for compatible model weights. Distinct from an API gateway that forwards calls to proprietary hosted models.
This entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.
- vLLM Production Stack deployment on Amazon EKS ↗
Vendor / upstream documentation · Reviewed 2026-11-12. Vendor support commitment: not assessed; confirm the selected release and contract.
The project publishes Amazon EKS provisioning and vLLM deployment scripts in its versioned Production Stack guide. Verify current chart versions, model licenses, accelerator availability, GPU quotas and storage; this is customer-hosted inference, not a hosted model API.
Cloud evidence reviewed 2026-11-12. Full listing and qualifications
Deployment and procurement checks
- Private offers
- AWS Marketplace private offers can contain negotiated pricing and terms for a buyer account. Confirm the product delivery model and your agreement; commitment credit is not established by a directory badge. Provider source ↗ · Reviewed 2026-11-14
- Private connectivity
- PrivateLink provides private endpoints to supported services and resources across VPC/account boundaries. Confirm that the selected product exposes a compatible endpoint service. Provider source ↗ · Reviewed 2026-11-14
- Cross-account operations
- Use an IAM role with scoped permissions and a third-party external ID where applicable. Review the trust principal, support access and revocation separately from installation access. Provider source ↗ · Reviewed 2026-11-14
- Government environments
- AWS GovCloud (US) is a distinct isolated-region environment. A commercial-region product deployment does not establish availability or certification there. Provider source ↗ · Reviewed 2026-11-14
For every provider, confirm marketplace/private-offer terms, commitment eligibility, regional service availability, private connectivity and any vendor access against your selected deployment. Provider capabilities do not automatically establish application support.
Plan this deployment
Account boundary: AWS account. Access: IAM roles and policies. Network: VPC.
Provider documentation ↗ · Guidance reviewed 2026-11-10. Confirm the application edition, region, support access, backup locations and operating responsibilities before choosing.
Compare this category across clouds · Search these listings · Compare 2–4 listings