AI inference & gateways
Model-serving runtimes, inference gateways and provider routing.
5 listings, including 5 from other publishers. Entries cover the deployment options documented on their individual pages.
Read this page as JSON or Markdown.
Watch this category with RSS · Build a requirements shortlist
Deployment comparison
| Listing | Documented clouds | Deployment and infrastructure | Licensing | Installation | Operations and limitations |
|---|---|---|---|---|---|
| LiteLLM Proxy Self-Hosted | AWS, Azure, Google Cloud, Railway, Render | Customer-operated Servers / containers | Open Source / Open Core MIT outside the enterprise directory; enterprise code and features have separate license terms. | Official container image, Docker Compose or Kubernetes deployment; production infrastructure examples are documented. | Customer manages gateway availability, credentials, database, access policies, logs and upgrades, including pinning versions and verifying release artifacts with the documented version-appropriate Cosign method; model providers remain separate services.Scope and evidenceThe gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.
Cloud evidence reviewed 2026-11-12 |
| LocalAI | AWS | Customer-operated Servers / containers | Open Source / Open Core MIT License for upstream LocalAI. Model artifacts and inference backends can have different license and hardware requirements. | Run official CPU/GPU Docker images or the documented binaries and configure models, storage and endpoint access. | Customer manages models, backends, hardware capacity, API authentication, persistent storage and upgrades.Scope and evidenceCPU availability does not guarantee acceptable performance for every model. Model galleries and artifact downloads contact external registries; optional integrations and remote backends can change data boundaries.
Cloud evidence reviewed 2026-11-12 |
| NVIDIA AI Enterprise | AWS, Azure, Google Cloud, Oracle Cloud, Alibaba Cloud, Tencent Cloud, Volcano Engine | Customer-operated Servers / containers | Commercial Only Commercial NVIDIA AI Enterprise subscription, consumption or perpetual license; most customer-cloud paths require BYOL. | NVIDIA AI Enterprise or GPU-optimized cloud VM images, standard instances with BYOL, or managed Kubernetes on supported cloud providers | Customer manages cloud GPU instances or Kubernetes, drivers, GPU Operator and container runtime, networking, storage, identities, NGC access, model services, monitoring, upgrades and license activation.Scope and evidenceThis entry covers customer-operated NVIDIA AI Enterprise software, not a managed AI service. A license is required and is not included with most standard, GPU-optimized VMI or managed-Kubernetes deployments; on-demand NVIDIA AI Enterprise VMIs and private offers use separate included-license processes. Not every component is available on every cloud or deployment model. GPU deployments are generally licensed per GPU, while CPU-only instances require an instance license. Tencent Cloud and ByteDance Volcano Engine have documented standard-instance BYOL paths. These guides require certified software and supported instance types; they do not establish that every GPU, component or provider image is eligible. Their per-instance wording differs from the general per-GPU licensing guidance, so confirm the applicable licensing unit. The November lifecycle matrix lists active 9.0/R620, 8.3/R595, and 7.9 LTSB/R580 branches with different end dates. NeMo Microservices support ended October 1, 2026, after 25.12 became the final GA release; select and patch each component branch independently.
Cloud evidence reviewed 2026-11-16 |
| SGLang | Google Cloud | Customer-operated Servers / containers | Open Source / Open Core Apache License 2.0 for upstream SGLang. Model weights and hardware/runtime dependencies have their own licenses. | Install the package or use official containers; the reviewed primary installation path targets Linux with supported NVIDIA GPUs and compatible drivers. | Customer manages accelerator capacity, model artifacts, runtime compatibility, endpoint access, including isolation of debug and dumper routes, monitoring, and upgrades.Scope and evidenceModel downloads may require external registries and gated-model credentials. Pin compatible model/runtime/image versions and review model-specific permissions. Generic container instructions do not establish deployment badges for every hyperscaler. CVE-2026-111391 rates SGLang 0.5.10 through 0.5.22 critical (CVSS v4.0 9.2) for Python code execution through the debug dumper filter. The upstream report demonstrates the shared-API path with DUMPER_SERVER_PORT=reuse: an ordinary API-key holder can reach it when no separate admin API key is configured; access can be anonymous when no API key is configured. With an admin API key configured, that key is required. Exploitation also requires access to the dumper configuration and a subsequent model execution that reaches a dumper hook. Upstream issue #44188 remains open, and the v0.5.23 tag still contains the eval-based filter path even though that tag is outside the CVE record's enumerated affected range. Do not enable or expose the debug dumper endpoint; restrict it at the network layer and verify an upstream fix before treating a later version as remediated.
Cloud evidence reviewed 2026-11-12 |
| vLLM Self-Hosted | AWS, Azure, Google Cloud | Customer-operated Servers / containers | Open Source / Open Core Apache 2.0 for vLLM; model weights and supporting dependencies have independent licenses and access conditions. | Official vLLM Docker image or supported Python installation; choose a compatible hardware/backend configuration. | Customer manages model acquisition, hardware capacity, endpoint protection, availability, upgrades and model/runtime compatibility.Scope and evidenceThis entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.
Cloud evidence reviewed 2026-11-12 |
Choose 2–4 listings to compare
Browse listing cards
AI inference & gateways +1 more
LiteLLM Proxy Self-Hosted
Self-hosted AI API gateway providing model routing, access controls and usage tracking.
- AWS
- Azure
- Google Cloud
- Railway
- Render
Buyer facts and open questions →
10 sources checked
AI inference & gateways +1 more
LocalAI
Local model inference service exposing configurable AI capabilities through compatible API endpoints.
- AWS
Buyer facts and open questions →
3 sources checked
AI tools & infrastructure +1 more
NVIDIA AI Enterprise
Commercial enterprise AI software suite deployed on customer cloud GPU instances or managed Kubernetes clusters.
- AWS
- Azure
- Google Cloud
- Oracle Cloud
- Alibaba Cloud
- Tencent Cloud
- Volcano Engine
Buyer facts and open questions →
10 sources checked
AI inference & gateways +1 more
SGLang
Model-serving framework for running supported language and multimodal models on customer-controlled compute.
- Google Cloud
Buyer facts and open questions →
7 sources checked
AI inference & gateways +1 more
vLLM Self-Hosted
Self-hosted model inference and serving engine with OpenAI-compatible APIs.
- AWS
- Azure
- Google Cloud
Buyer facts and open questions →
5 sources checked
How these listings fit
- LiteLLM Proxy Self-Hosted
- Self-hosted model API gateway for routing, virtual keys, spend tracking and policy enforcement. It routes inference requests; it is not an inference engine for all supported models.
- LocalAI
- Local inference service with OpenAI-compatible API endpoints and configurable model backends; compatible API support does not require a hosted OpenAI service.
- NVIDIA AI Enterprise
- Includes supported inference and model-serving components for deploying generative-AI and machine-learning workloads.
- SGLang
- Model-serving framework for LLM and multimodal inference; an inference runtime rather than a full multi-provider governance gateway.
- vLLM Self-Hosted
- Model inference engine and OpenAI-compatible serving API for compatible model weights. Distinct from an API gateway that forwards calls to proprietary hosted models.
Review dates and sources appear on each listing. Functionality, edition, deployment and licensing must be checked together. Filter this category with cloud and deployment controls →