how to deploy llms in production: the requirements every engineering team must get right

why deploying llms is a different beast

getting a large language model (llm) to answer questions inside a notebook is easy. getting that same model to serve thousands of real users, around the clock, without crashing or burning your budget is a completely different challenge. this is the gap between "it works on my machine" and true production readiness — and it is exactly where most beginner projects struggle.

the good news? if you already have some coding experience and a basic understanding of devops, you are closer to success than you think. deployment is a set of learnable requirements, and this article walks you through each one.

what makes llm deployment different from traditional apps

a typical web app calls a database, gets a small json response, and returns it. llm applications behave very differently:

• heavy hardware needs: models must fit in gpu memory, which is expensive and limited.
• slow responses: generating hundreds of tokens takes seconds, not milliseconds.
• unpredictable output length: one user may ask for a one-line answer, another for a 2,000-word essay.
• stateful conversations: you often need to track chat history across requests.
• non-deterministic output: the same prompt can return different answers, which complicates testing.

because of this, deploying an llm is a full stack problem: the model layer, the api layer, the infrastructure layer, and the monitoring layer all have to work together. let's break down every requirement.

requirement 1: choose your hosting strategy first

before writing a single line of deployment code, decide where your model will live. this choice affects cost, latency, privacy, and complexity more than anything else.

option a: use a managed llm api

services like openai, anthropic, or cloud provider apis (aws bedrock, azure openai, google vertex ai) host the model for you. you simply send http requests.

• pros: no gpus to manage, fast to launch, always up to date.
• cons: pay-per-token costs grow quickly at scale, and your data leaves your infrastructure.

option b: self-host an open-source model

models like llama 3, mistral, or qwen can run on your own servers using tools such as vllm or ollama.

• pros: full control over data, predictable costs at high volume, no rate limits from a vendor.
• cons: you handle gpu provisioning, updates, and failures yourself.

quick decision guide

• prototype or low traffic? start with a managed api. ship first, optimize later.
• strict data privacy (healthcare, finance, internal tools)? self-host.
• millions of requests per month? do the math — self-hosting often becomes cheaper per token.
• unsure? use a hybrid approach: a small self-hosted model for simple tasks, a large api model for complex ones.

requirement 2: size your infrastructure correctly

the most common beginner mistake is underestimating gpu memory. here is a simple rule of thumb for a model loaded in fp16 precision:

required gpu memory (gb) ≈ number of parameters (in billions) × 2

so an 8b-parameter model needs roughly 16 gb just for the weights — and that is before adding memory for the kv cache (the model's short-term memory during generation) and activation overhead. in practice, you want 20–30% headroom.

• 7b–8b model: one 24 gb gpu (e.g., rtx 4090, l4, a10g) works well.
• 13b–14b model: 40–48 gb (e.g., a100 40gb with quantization, or a100 80gb).
• 70b model: multiple high-end gpus or aggressive quantization (int8/int4).

pro tip: quantization (reducing precision from fp16 to int8 or int4) can cut memory needs by 2–4× with modest quality loss. it is one of the highest-impact optimizations a small team can make.

also do not forget networking and storage: model weights can be tens of gigabytes, so use fast disks and keep a local model cache so deployments do not re-download everything.

requirement 3: pick a proper serving engine

never serve a model with a plain pytorch script in production. modern serving engines handle batching, memory management, and concurrency for you. popular choices include vllm, text generation inference (tgi), and ollama for local development.

starting vllm with a production-ready configuration takes one command:

pip install vllm

vllm serve meta-llama/llama-3-8b-instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90 \
  --port 8000

this exposes an openai-compatible api at /v1/chat/completions, which makes integration painless.

build a simple api wrapper (where full stack coding shines)

in production, clients should never talk to the model server directly. wrap it in your own api so you can validate input, enforce limits, and swap models later without breaking anything:

from fastapi import fastapi, httpexception
from pydantic import basemodel
import httpx

app = fastapi()

class promptrequest(basemodel):
    prompt: str
    max_tokens: int = 512

@app.post("/generate")
async def generate(request: promptrequest):
    # basic input validation protects the model and your budget
    if not request.prompt.strip():
        raise httpexception(status_code=400, detail="prompt is empty")
    if len(request.prompt) > 8000:
        raise httpexception(status_code=400, detail="prompt too long")

    async with httpx.asyncclient(timeout=120) as client:
        response = await client.post(
            "http://localhost:8000/v1/chat/completions",
            json={
                "model": "meta-llama/llama-3-8b-instruct",
                "messages": [{"role": "user", "content": request.prompt}],
                "max_tokens": request.max_tokens,
            },
        )
        response.raise_for_status()
    return response.json()

this small layer is your safety net. every serious llm deployment has one.

requirement 4: containerize everything (devops 101)

if your team practices devops, you already know the answer: ship your service as a docker container. containers make your deployment reproducible, portable, and easy to roll back.

from python:3.11-slim

workdir /app

copy requirements.txt .
run pip install --no-cache-dir -r requirements.txt

copy . .

expose 8080

# add a health check endpoint and run with multiple workers
healthcheck --interval=30s cmd curl -f http://localhost:8080/health || exit 1
cmd ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080", "--workers", "2"]

key points to remember:

• never bake model weights into the image if they are huge — download them at startup from a model registry or object storage.
• use environment variables for all configuration (model name, ports, thresholds) so the same image works in staging and production.
• add a /health endpoint so your orchestrator (kubernetes, docker compose, ecs) knows when the service is ready.

requirement 5: add observability before you scale

you cannot fix what you cannot see. llm systems fail in subtle ways — slow responses, silent timeouts, degraded quality — so instrument everything from day one.

the metrics that actually matter

• time to first token (ttft): how long users wait before seeing any output. this drives perceived speed more than total time.
• tokens per second: your generation throughput under load.
• error rate and timeouts: the percentage of failed requests.
• gpu utilization and memory: warn before the kv cache pushes you out of memory.
• prompt and completion token counts: these translate directly into cost.

start with structured logging — it takes ten minutes and saves days of debugging:

import logging, time

logger = logging.getlogger("llm-service")

async def generate_with_metrics(prompt: str):
    start = time.time()
    result = await call_model(prompt)
    duration = round(time.time() - start, 2)

    logger.info("inference_done", extra={
        "duration_s": duration,
        "prompt_chars": len(prompt),
        "output_tokens": result["usage"]["completion_tokens"],
    })
    return result

later, pipe these logs into prometheus and grafana, or use an llm-specific tool like langfuse or opentelemetry traces. if your llm also powers public-facing pages, remember that slow responses hurt user experience and your seo, since search engines favor fast, reliable sites.

requirement 6: control costs with rate limiting and caching

llm inference is one of the most expensive operations you will ever run per request. two techniques keep the bill under control:

1. rate limiting

protect your gpus from runaway scripts and abusive users with a simple per-user limit:

import time
from collections import defaultdict

rate_limit = 10          # requests allowed
window_seconds = 60
request_log = defaultdict(list)

def allow_request(user_id: str) -> bool:
    now = time.time()
    request_log[user_id] = [
        t for t in request_log[user_id] if now - t < window_seconds
    ]
    if len(request_log[user_id]) >= rate_limit:
        return false
    request_log[user_id].append(now)
    return true

2. response caching

many real-world prompts repeat — faq answers, similar support tickets, standard code questions. cache them:

import hashlib, json

def cache_key(prompt: str, max_tokens: int) -> str:
    payload = json.dumps({"prompt": prompt, "max_tokens": max_tokens})
    return hashlib.sha256(payload.encode()).hexdigest()

# check redis/cache first; only call the model on a cache miss.
# exact-match caching is easy; "semantic" caching (matching similar
# meanings) is a powerful next step once traffic grows.

even a modest cache hit rate of 20–30% can cut your inference bill dramatically.

requirement 7: secure your endpoint from day one

an unauthenticated gpu endpoint is an open invitation to abuse. at minimum, implement:

• authentication: require api keys or jwt tokens on every request.
• secrets management: store keys in environment variables or a vault — never hardcode them in your code or commit them to git.
• input validation: cap prompt length, strip control characters, and reject malformed payloads.
• output filtering: screen responses if your product is user-facing, especially for harmful or off-topic content.
• network isolation: keep the model server on an internal network; only your api wrapper should be public.

a minimal auth dependency in fastapi looks like this:

from fastapi import depends, header, httpexception
import os

valid_keys = set(os.getenv("api_keys", "").split(","))

async def verify_api_key(x_api_key: str = header(...)):
    if x_api_key not in valid_keys:
        raise httpexception(status_code=401, detail="invalid api key")
    return x_api_key

@app.post("/generate", dependencies=[depends(verify_api_key)])
async def generate(request: promptrequest):
    ...

requirement 8: automate deployment with ci/cd

manually copying code to a server works for a weekend project, but a production llm service needs a ci/cd pipeline — a core devops practice. it should run tests, build the container, and deploy automatically:

name: deploy llm service
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: run tests
        run: pytest tests/ -v

      - name: build and push docker image
        run: |
          docker build -t myregistry/llm-api:${{ github.sha }} .
          docker push myregistry/llm-api:${{ github.sha }}

      - name: roll out new version
        run: |
          ssh ${{ secrets.server_user }}@${{ secrets.server_host }} \
            "docker pull myregistry/llm-api:${{ github.sha }} && \
             docker stop llm-api || true && docker rm llm-api || true && \
             docker run -d --name llm-api -p 8080:8080 \
               --gpus all \
               myregistry/llm-api:${{ github.sha }}"

two habits that will save you many late nights:

• always keep the previous model version available so you can roll back instantly if quality drops.
• test model changes on a small traffic slice (canary deployment) before full rollout.

a pre-launch checklist for engineering teams

before you send real users to your llm service, verify every item below:

• model license permits commercial use in your region.
• gpu memory sized with at least 20–30% headroom for the kv cache.
• api wrapper with input validation and timeouts in place.
• authentication and rate limiting enabled.
• docker image builds cleanly, with health checks passing.
• ci/cd pipeline tested, including a rollback path.
• dashboards show latency, tokens per second, errors, and cost.
• load test performed with prompts at your maximum expected length.
• secrets stored outside the codebase.

common beginner mistakes (and how to avoid them)

• testing only with short, friendly prompts. users will paste 10,000-character documents. test with worst-case inputs.
• ignoring kv cache growth. long conversations consume more memory per request than your initial benchmark suggested.
• no request timeouts. one stuck generation can hold a worker forever. always set timeouts on both sides.
• hardcoding the model name everywhere. pass it via environment variables so switching models is a config change, not a refactor.
• skipping load testing. a single-user benchmark tells you almost nothing about behavior with 50 concurrent requests.

final thoughts: you are closer than you think

deploying llms in production may sound intimidating, but every requirement in this article is a well-understood engineering problem with mature tools behind it. start small: wrap a model in an api, containerize it, add logging and rate limits, and automate the deploy. each step builds real full stack and devops skills that transfer to every system you will build in your career.

perfect reliability is not the goal on day one — visibility, safety rails, and steady iteration are. ship version one, watch your metrics, improve continuously, and before long you will be the engineer on your team who knows exactly how to take models from notebook to production. happy coding!