How to host open-source LLMs: from Ollama to Kubernetes

Sensitive data that must not leave the company. An API invoice growing faster than the product. Or simply the desire to understand what’s happening under the hood. The reasons for running your own language model keep multiplying — and the good news is that today it’s easier than ever. In this guide we walk the whole path: choosing a model, quantisation, hardware requirements, the three most used inference engines, renting GPUs in the cloud and scaling on Kubernetes.

When self-hosting makes sense

• Privacy and compliance — data doesn’t leave your infrastructure. For healthcare, finance or internal know-how often the only viable path.
• Costs at high volume — an API is cheap to start, but at millions of tokens a day your own GPU pays for itself. Count honestly, though: add electricity, administration and your time to the price of the card.
• Control and stability — the model won’t change or disappear under your hands, you can fine-tune it and tune inference (see the mitigation of the U-curve from the previous article).
• Offline and edge scenarios — an assistant running without internet, on a laptop or in a closed network.

Conversely, if you need the top quality of frontier models and your volumes are small, stay with an API — open-source models are excellent, but they match the biggest commercial ones only in narrower domains.

Choosing a model

The offer on Hugging Face changes every month, but the families remain: Llama (Meta), Qwen (Alibaba), Mistral, Gemma (Google), DeepSeek or GPT-OSS (OpenAI). When choosing, consider three things:

1. Size — given in billions of parameters (7B, 14B, 70B…). Bigger = smarter, but slower and hungrier for memory. Watch out for MoE (mixture-of-experts) models: with them the total number of parameters differs from the number active per token — you have to fit the total into memory, in speed they correspond to the active ones.
2. Variant — you want the instruct version (fine-tuned for conversation and instructions), not base. For agentic use verify tool calling support, for RAG a sufficient context window.
3. Licence — Apache 2.0 and MIT have no catches; community licences (e.g. Llama) tend to have conditions for commercial deployment. Always read them.

Practical advice: don’t start with the question “what’s the biggest model I can run” but “what’s the smallest model that can handle my task”. For classification, extraction or summarisation a 4–8B model is often enough, running fast and cheaply; you won’t make a universal assistant out of it, though. Only evaluation on your data will decide.

Quantisation: the key to reasonable hardware

A model in full precision (FP16/BF16) needs roughly 2 bytes per parameter — a 7B model thus ~14 GB of VRAM just for the weights. Quantisation rounds the weights to fewer bits: with 4-bit quantisation consumption drops to roughly a quarter, with a small loss of quality acceptable for most uses. The loss isn’t zero, though, and grows with the aggressiveness of the quantisation — below 4 bits it tends to be noticeable, and for tasks sensitive to precise reasoning (maths, code) even earlier.

• GGUF — the format of the llama.cpp/Ollama ecosystem, levels Q2 to Q8. A reasonable compromise is Q4_K_M; Q5_K_M or Q8_0 give higher quality. It can run on CPU, GPU or hybrid.
• AWQ / GPTQ — 4-bit formats optimised for GPU inference, typically for vLLM. AWQ protects “important” weights with higher precision, so it holds quality a bit better.
• FP8 — on newer cards (Hopper, Ada and newer) the golden middle way for production: half the memory compared to FP16 with practically unmeasurable loss.

To the weights, though, you have to add the KV cache — the memory in which the model stores attention keys and values for already processed tokens. It grows linearly with context length and with the number of concurrent requests; on a server with long conversations it can easily swallow more than the weights themselves. That is exactly why production engines devote so much attention to managing it.

Take this as an indicative table — actual consumption depends on context length and the number of concurrent requests. Models that don’t fit on one card can be spread across multiple GPUs (tensor parallelism), at the price of demands on the interconnect between cards. And don’t forget that thanks to llama.cpp smaller models can be run purely on CPU too, just significantly more slowly.

Three tools, three scenarios

Ollama — the fastest start

Ollama wraps llama.cpp in a Docker-style tool: with one command it downloads the model, runs it quantised and exposes a local API. It handles memory management, automatic fallback to CPU and unloading of unused models itself. Ideal for development, prototypes and personal use:

# instalace (Linux), pak staci:
ollama run qwen3:8b

# model bezi a zaroven je dostupny na http://localhost:11434

llama.cpp — maximum control and the broadest hardware

llama.cpp is the C engine Ollama is built on. It runs on CUDA, AMD, Apple Silicon and plain CPU and gives you full control over quantisation, offloading layers to the GPU and other parameters. It also includes a server with an OpenAI-compatible API:

llama-server -m qwen3-8b-Q4_K_M.gguf --port 8080 -ngl 99
# -ngl 99 = vsechny vrstvy na GPU; snizte, kdyz se model do VRAM nevejde

vLLM — production serving for multiple users

As soon as the model is to serve a team or an application with dozens of concurrent requests, it’s time for vLLM. Its two key technologies:

• PagedAttention — the KV cache is managed in small pages, similar to virtual memory in an operating system. Fragmentation disappears and significantly more concurrent requests fit into the same VRAM.
• Continuous batching — new requests jump into a running batch as soon as a slot frees up; the GPU doesn’t stall waiting for the slowest request.

The result: under concurrent load, throughput an order of magnitude higher than Ollama or llama.cpp, at the price of a more complex setup and the requirement for an NVIDIA GPU. For a single user it’s needlessly heavy infrastructure; for a team or a production application, on the other hand, the right tool.

pip install vllm
vllm serve Qwen/Qwen3-8B-AWQ --max-model-len 16384

# pozor: vLLM si standardne zabere ~90 % VRAM pro KV cache;
# pokud kartu sdilite, pridejte --gpu-memory-utilization 0.5

One API for everything

All three tools expose an OpenAI-compatible endpoint. That means you write the application once and switch between a local model, your own server and a commercial API just by changing the URL:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",   # Ollama; u vLLM napr. :8000/v1
    api_key="neni-potreba",
)

odpoved = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Vysvetli mi kvantizaci LLM."}],
)
print(odpoved.choices[0].message.content)

No GPU? Rent one

Your own card isn’t a requirement — between “everything at home” and “commercial API” there is a whole spectrum of online platforms. They differ mainly in how much administration they leave to you:

• GPU clouds (IaaS) — you rent a bare machine with a GPU by the hour and install vLLM or Ollama yourself. Typically RunPod, Vast.ai (a GPU marketplace with the lowest prices), Lambda or the European CoreWeave; AWS, Azure and Google Cloud of course have GPU instances too, usually at higher prices. Maximum control, full responsibility.
• Serverless GPU — you upload a container or a function and pay only for the seconds the inference actually runs; scaling to zero is handled by the platform. E.g. Modal, Replicate or RunPod Serverless. Great for bursty load; the price is the cold start (loading weights into VRAM takes tens of seconds).
• Managed inference of open models — a ready-made API over open-source models where you deal with nothing at all: Together AI, Fireworks AI, Groq (extremely fast inference on custom chips) or Hugging Face Inference Endpoints. Technically this is no longer self-hosting, but you keep the freedom to choose the model and an escape route: thanks to the OpenAI-compatible API you can later move to your own hardware by changing a single URL.

A typical project evolution: a prototype on a laptop with Ollama, a pilot on a rented RunPod with vLLM, and only when sustained load is clear, an investment in your own hardware — or a move to Kubernetes.

Larger scale: Kubernetes

A single server with vLLM serves surprisingly much. But as soon as you need multiple replicas for availability, multiple models side by side, autoscaling by load and zero-downtime deployment of new versions, you arrive where the industry has standardised on Kubernetes. From the K8s point of view LLM inference is “just” another workload — with three specifics that need solving:

1. GPU as a schedulable resource — nodes with cards require the NVIDIA GPU Operator (drivers, device plugin, monitoring). A Pod then requests a GPU in resources.limits as nvidia.com/gpu: 1. Cards can also be shared between smaller workloads (MIG, time-slicing).
2. Slow starts — tens of GB of weights have to get into VRAM when the pod starts. This is addressed with a model cache on a persistent volume or in a registry, readiness probes (the pod doesn’t receive traffic until the model is loaded) and conservative scale-down so the cluster doesn’t needlessly recycle replicas.
3. Smarter load balancing — classic round-robin is unsuitable for LLMs: requests differ in length by orders of magnitude and it pays to route them according to the KV cache utilisation and prefix-cache hits of specific replicas.

A minimal vLLM deployment on K8s fits into an ordinary Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen3
spec:
  replicas: 2
  selector:
    matchLabels: { app: vllm-qwen3 }
  template:
    metadata:
      labels: { app: vllm-qwen3 }
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "Qwen/Qwen3-8B-AWQ",
                 "--max-model-len", "16384"]
          ports: [{ containerPort: 8000 }]
          resources:
            limits: { nvidia.com/gpu: 1 }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
          volumeMounts:
            - { name: model-cache, mountPath: /root/.cache/huggingface }
      volumes:
        - name: model-cache
          persistentVolumeClaim: { claimName: hf-cache }

For the higher floors there are specialised layers: KServe (standardised InferenceService CRD, scale to zero), vLLM Production Stack (a Helm chart with a router that routes by KV cache) or llm-d — Kubernetes-native distributed inference that separates the prefill and decode phases onto different pods. Autoscaling is then driven not by CPU but by LLM metrics (queue depth, KV cache utilisation) via KEDA or an HPA with custom metrics.

What to watch out for in production

If you also run a planning loop, tools and memory around the model, it’s no longer just an inference server. In the follow-up article How to host AI agents on AWS we compare Lambda, ECS Fargate and AgentCore.

• Securing the endpoint — the local API has no authentication. Before exposing it to the network it belongs behind a reverse proxy with API keys and TLS; on K8s this is handled by ingress and NetworkPolicy.
• Concurrency and queues — Ollama and llama.cpp aren’t built for dozens of parallel users; that’s what vLLM (or SGLang or TGI) is for.
• Monitoring — watch time to first token (TTFT), tokens/s, queue depth and VRAM utilisation; vLLM exports metrics for Prometheus, the DCGM exporter serves GPU metrics.
• Quality evaluation — a quantised smaller model may not be enough for your task. Before deployment compare the outputs with a reference API model on your own data — and repeat the measurement with every change of model or quantisation.
• Costs continuously — the hourly GPU rate is only part of the bill; an unused reserved card is the most expensive card. Autoscaling and scale-to-zero save the most for bursty load.

About the author

Zdeněk Vais

Founder · Senior Principal Software Engineer, Oracle

Zdeněk has been programming since high school. Since then he has gone from programmer through team leader to architect, building large financial systems, scientific applications and IoT solutions for aircraft along the way. He currently works as a senior principal software engineer at Oracle, where he builds the cloud for media services.

Meet our team

← Back to the blog