# How do I run ChatGPT-style AI privately in my own cloud?

Canonical page: https://launchinyourcloud.com/questions/private-chat-ai/

Combine a self-hosted interface such as [Open WebUI](/directory/open-webui/) with inference such as [vLLM](/directory/vllm/), or a gateway such as [LiteLLM](/directory/litellm/) routing to approved endpoints. Hosting the interface does not host the model. Review inference, embeddings, storage and logs separately.

Answer page: https://launchinyourcloud.com/questions/private-chat-ai/
Answer reviewed: 2026-11-21

- [Private AI interface and inference stack](/stacks/private-ai-interface-and-inference/)
- [AI applications and RAG directory](/categories/ai-applications-rag/)


## Map every AI layer

- Identify where prompts, attachments, embeddings, vector data, responses and conversation history are processed and retained.
- Choose the actual inference endpoint and model license, and size the required CPU, GPU, memory and storage for that model and workload.
- Review gateway routes, credentials, plugins, web access, logs and observability exports. Test egress rather than inferring isolation from the user interface.

## Start with a documented connection

The private AI stack page documents an Open WebUI and vLLM connection. A gateway is an alternative routing layer, not proof of local inference. A cloud badge for one component does not establish that all layers, models or optional services run on the same provider or can operate offline.

## Reviewed listing references

These are illustrative candidates, not a ranking or a feature-parity claim.

### LiteLLM Proxy Self-Hosted

Self-hosted AI API gateway providing model routing, access controls and usage tracking.

Operations: Customer manages gateway availability, credentials, database, access policies, logs and upgrades, including pinning versions and verifying release artifacts with the documented version-appropriate Cosign method; model providers remain separate services.
License: MIT outside the enterprise directory; enterprise code and features have separate license terms.
Scope: The gateway forwards each request to its configured model provider. Self-hosting the proxy does not ensure prompts stay inside the account: external model APIs, guardrail services and logging integrations can receive data. Use appropriate self-hosted inference endpoints and explicitly review outbound integrations when required. For release integrity, release images and the Helm chart starting with v1.111.0-dev.3 use keyless Cosign signatures bound to LiteLLM's release workflow; earlier releases require the documented historical public keys. The October 21 incident affected PyPI v1.107.0 and Docker Hub/GHCR v1.107.0 plus mutable latest, main-latest and main-stable tags during the documented exposure windows. Check the incident's artifact digests and remediation instructions; the compromised images carried valid historical signatures, so signature verification alone does not establish safety. LiteLLM v1.110.0 changes authentication, guardrail, accounting, and S3 behavior. The 1.108.11 LTS patch also fixes Bedrock guarded_text handling and restricts os.environ team settings to proxy administrators; review both the selected release line and existing incident-response or image-signing guidance. LiteLLM v1.108.12 LTS fixes tool_policy allowing blocked /v1/messages tool names longer than 64 characters. The standard v1.110.1 UI image changes to slim nginx and removes curl while retaining wget and nc, so retest health probes and scripts that assume curl.
Listing and sources: https://launchinyourcloud.com/directory/litellm/
Reviewed: 2026-11-19

### Open WebUI Self-Hosted

Self-hosted AI chat and knowledge interface for local model servers and cloud APIs.

Operations: Customer runs application storage, authentication, provider configuration, updates and backups.
License: Open WebUI license based on BSD-style terms with additional branding restrictions and defined exceptions; enterprise terms may be required for rebranding.
Scope: The UI sends messages to the model endpoint selected for a conversation. Local Ollama or vLLM endpoints can keep inference on customer infrastructure; cloud APIs, embeddings, tools and plugins have separate data flows. The current license restricts branding removal in larger deployments, with small-deployment, permission and enterprise-license exceptions; review the actual terms. Open WebUI v0.13.4 replaces Direct Connections and Integrations with Tether and migrates image settings. Deployments using Terminals server connections require Terminals 0.4.0 or later; upgrading that server from 0.3.x requires manual migration. Open Terminal and Tether terminal connections are unaffected by this migration. Tether introduces user-device and local-command paths that require explicit trust review.
Listing and sources: https://launchinyourcloud.com/directory/open-webui/
Reviewed: 2026-11-18

### vLLM Self-Hosted

Self-hosted model inference and serving engine with OpenAI-compatible APIs.

Operations: Customer manages model acquisition, hardware capacity, endpoint protection, availability, upgrades and model/runtime compatibility.
License: Apache 2.0 for vLLM; model weights and supporting dependencies have independent licenses and access conditions.
Scope: This entry covers the inference server, not a hosted model service. It can run supported model weights on customer infrastructure, but those weights may need external downloads or access approval and are not licensed by vLLM. Review default usage telemetry and optional outbound integrations; hosting inference does not automatically create an air-gapped system. vLLM v0.34.0 removes Model Runner V1; configurations setting VLLM_USE_V2_MODEL_RUNNER=0 now fail at startup. It changes the default block size from 16 to 64, enables extensible KV cache by default with gpu_memory_utilization defaulting to 1.0, drops ROCm 7.2 wheels and images in favor of ROCm 10.0, requires huggingface_hub 2.0.0 or later, and removes QuTLASS/fp_quant and Marlin W4A8 support. Review the upstream breaking-change table and test model, quantization, memory, and accelerator compatibility before rollout.
Listing and sources: https://launchinyourcloud.com/directory/vllm/
Reviewed: 2026-11-17

