DevOps12 min readMay 4, 2026

Deploying LLM Inference on Kubernetes with vLLM

B

BADJO Dibéa Koffi

Published on May 4, 2026

The Cost Problem

A 13B model needs 26GB VRAM. An A100 on AWS costs $2,952/month 24/7. But most apps don't need 24/7 inference — traffic is bursty.

vLLM

vLLM uses PagedAttention for 2-4x higher throughput than HuggingFace. It exposes an OpenAI-compatible API.

FROM vllm/vllm-openai:latest
ENV MODEL_NAME=meta-llama/Llama-2-13b-chat-hf
CMD ["python", "-m", "vllm.entrypoints.openai.api_server", \
     "--model", "$MODEL_NAME", "--gpu-memory-utilization", "0.90"]

KEDA Autoscaling

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  scaleTargetRef:
    name: llm-inference
  minReplicaCount: 0
  maxReplicaCount: 3
  triggers:
    - type: prometheus
      metadata:
        query: sum(vllm_num_requests_waiting)
        threshold: "5"