Self-hosted AI: when your company legally can't use ChatGPT

Published September 15, 2025

"Just use the API" is the default answer to almost every AI question right now. For a lot of companies it's also not an option. Not because of budget, not because of taste — because legal won't sign off, or a regulator won't let them, or a client contract explicitly forbids it.

If you're an engineer or CTO who's hit that wall, this is for you. I'll go through why the wall exists, what a realistic self-hosted stack looks like today, what you give up by going that route, and when a compliant cloud option is actually the smarter move instead.

Why "just call OpenAI" isn't always on the table

The block is usually one of a few things, and they stack.

Data residency. Some contracts, some public-sector rules, and some national regulations require data to stay within a specific jurisdiction, on infrastructure you control or can audit. Sending customer records to a US-based API endpoint — even one with an EU region — can fail that test depending on how the requirement is written.

GDPR and processor chains. Every external API call is a new data processor in your chain. That means a Data Processing Agreement, a documented legal basis, and — for anything involving special category data (health, biometric, etc.) — a lot more scrutiny. Some companies decide it's simpler to have zero external processors for a given data class than to manage the paperwork for one more vendor.

Legal privilege. Law firms and in-house legal teams dealing with privileged material have a specific problem: privilege can be waived by disclosure to a third party. Whether sending a privileged document to an LLM API counts as that kind of disclosure is a live legal question, and a lot of firms aren't willing to be the test case.

Sector rules. Healthcare (HIPAA in the US, national health data rules elsewhere), finance (data localization, model risk management requirements for anything touching credit or trading decisions), defense and government contracting — all of these have existing frameworks that predate LLMs and don't have a clean "AI API" carve-out. Compliance teams default to no until someone proves yes is safe.

Board-level policy. Sometimes there's no specific regulation at all — just a risk committee that decided "no data leaves our infrastructure to any third-party AI service" as a blanket policy, because it's easier to enforce and audit than a nuanced per-use-case exception process. This is common after a security incident anywhere in the industry, not necessarily your own.

None of these are irrational. They're risk-averse, and the risk they're avoiding is real. The mistake is treating "we can't use the ChatGPT API" as "we can't use AI at all." There's a middle path.

What a real self-hosted stack looks like in 2025

Forget the idea that self-hosting means falling back to a toy model that can barely follow instructions. The open-weight ecosystem has genuinely caught up for a large share of practical tasks — not frontier-level reasoning, but summarization, extraction, classification, internal chat, code assistance on well-scoped tasks, and RAG over your own documents all work well with the current generation of open models.

The model layer. The realistic choices as of late 2025 cluster around three families: Llama (Meta), Qwen (Alibaba), and Mistral. Each ships a range of sizes, from small models that run on a single consumer-ish GPU up to dense or mixture-of-experts models that need real datacenter hardware. Pick based on task, not hype — a well-tuned mid-size model for structured extraction will often beat a bigger general-purpose one you haven't adapted to the task.

Serving. This is where most of the engineering actually is. Two tools dominate:

• vLLM — production-grade inference server, built for throughput. Handles continuous batching, PagedAttention for memory efficiency, and an OpenAI-compatible API out of the box. This is what you want if you have real concurrent load.
• Ollama — much simpler to run, great for a single team or a low-concurrency internal tool, weaker on throughput at scale. A good starting point, not always the right long-term production choice.

A minimal vLLM deployment looks roughly like this:

# docker-compose.yml — single-node vLLM serving a mid-size open model
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    command: >
      --model Qwen/Qwen2.5-32B-Instruct
      --gpu-memory-utilization 0.90
      --max-model-len 32768
      --api-key ${VLLM_API_KEY}
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

That's the easy part. Model name, memory settings — a few lines. The work is everything around it.

Gateway / proxy layer. You don't want every internal app calling the inference server directly. Put a gateway in front — something like LiteLLM's proxy mode — so you get one place to manage API keys, rate limits, per-team usage tracking, model routing (send this task to the small model, that one to the big one), and a consistent OpenAI-compatible interface regardless of which backend is actually serving the request. This also gives you a single point to swap models later without touching every app that calls it.

SSO and audit logging. This is the part self-hosted deployments skip early and regret later. If this system is replacing ChatGPT for regulated data, it needs to meet the same bar ChatGPT Enterprise would: SSO (SAML/OIDC) at the gateway, every request logged with user identity, prompt, and response retained per your actual retention policy, and access reviewed like any other internal system touching sensitive data. Compliance signed off on this specifically because it's auditable — don't ship the version that isn't.

What you give up

Be honest with the team about the trade before they find out the hard way.

Capability gap. The best open-weight models are good, but they are not consistently at frontier-API level on hard reasoning, long-context synthesis, or cutting-edge coding tasks. For a lot of internal use — drafting, summarizing, classifying, answering questions over internal docs — the gap doesn't matter. For your hardest problems, it might.

Maintenance burden. An API call has no infrastructure for you to maintain. A self-hosted stack has GPUs that fail, drivers that need updating, a serving layer that needs upgrading, models that need re-evaluating every time a new version drops, and an on-call rotation if it's actually load-bearing for the business. This is a real team responsibility, not a one-time setup task.

You're now the one keeping up. Frontier labs ship improvements weekly. Open-weight releases are less frequent and it's on you to evaluate, test, and roll out each one. Falling behind is easy if nobody owns it.

When self-hosting is genuinely the right call

Self-host when the constraint is real and structural: a legal opinion says data can't leave your infrastructure, a regulator has said the same in writing, or a client contract has a specific no-external-processing clause you can't renegotiate. In those cases, there's no cloud workaround — you build the on-prem stack or you don't do AI for that workload.

When a compliant cloud option is the better move

A lot of "we need self-hosted AI" requirements turn out, on closer reading, to be "we need our data handled a specific way" — which cloud providers can often satisfy without you buying and running GPUs.

Look at EU-region deployments from major providers (Azure OpenAI in an EU region, for instance) before assuming you need on-prem. Look at zero-data-retention API agreements — several frontier labs offer contractual terms where your prompts and outputs are not retained or used for training, which addresses a lot of the underlying worry directly. Check whether your actual regulatory requirement is "data residency in region X" (solvable in cloud) versus "data must never leave infrastructure we physically control" (not solvable in cloud, full stop).

Get the actual requirement in writing from legal or compliance before you commit engineering time to either path. "We should probably self-host to be safe" is not a requirement — it's a guess, and it's an expensive one to build against if it turns out a zero-retention agreement with an EU region would have satisfied the real constraint at a fraction of the engineering cost.

Self-hosting is the right tool for a specific, verifiable class of problem. Confirm you're actually in that class before you commit to running GPUs in production.