Your AI Demo Works. Will It Survive Production?

01 / A deployment is only the beginning.

Your demo accepts a prompt, calls a model, and returns an answer. Production adds competing users, slow providers, failed nodes, and releases that must be reversible. A running Pod proves a process started. It does not prove a customer can complete a task.

Our illustrative architecture uses a stateless API, a durable queue, independent workers, an external model provider, and a database for task status and results. Kubernetes runs the API and workers. Helm packages their desired configuration into a versioned release. This is a design guide, not a complete deployable chart or a measured customer case study.

Define success before choosing replicas: task completion within an agreed deadline, tolerable failures, recovery time, and acceptable data loss. These targets determine scaling, retention, backups, and who gets paged.

02 / Package the whole application with Helm.

A chart should express a repeatable contract between application code and runtime. Separate API and worker Deployments so each can scale independently. Promote the same immutable container digest through staging and production; environment values change capacity and endpoints, not the build.

Use Chart.yaml for chart and application versions, Chart.lock for dependency resolution, values.yaml for safe defaults, and values.schema.json to validate inputs. Store templates for Deployments, Service, ingress, HPA, PDB, ServiceAccount, NetworkPolicy, ConfigMap, and migration Jobs. Keep environment values separate. Never store credentials in values files.

Cluster prerequisites need explicit owners: ingress or Gateway controllers, TLS certificates, metrics adapters, secret integrations, and telemetry collectors. A chart does not automatically provide them. Render and validate manifests against the target Kubernetes APIs, inspect the diff, scan the image, and test permissions, admission policies, storage, and scheduling in staging.

03 / Healthy processes. Ready Pods. Healthy nodes.

Startup probes answer: has initialization finished? Give measured startup time enough headroom. Until a startup probe succeeds, Kubernetes holds back liveness and readiness checks.

Readiness probes answer: can this Pod accept work? Failure removes the Pod from normal Service traffic without restarting it. Check the conditions needed to serve requests. An API that can still accept durable queued work need not become unready just because the model provider is down.

Liveness probes answer: is the local process stuck in a way restarting can fix? Keep them local and cheap. A provider outage must not produce a restart storm. Implement real health endpoints and load-test their behavior.

Use multiple replicas, a Deployment rollout strategy, and a PodDisruptionBudget together. For an illustrative three-replica API, minAvailable: 2 permits one voluntary eviction at a time. A rollout with maxUnavailable: 0 and maxSurge: 1 requires spare capacity for another Pod.

A PDB constrains voluntary eviction requests. It cannot prevent involuntary node loss and does not govern a Deployment’s own rolling update. A budget permitting no evictions can block maintenance. Match its selector to the API Pod labels and test a node drain.

Monitor Node Ready, MemoryPressure, DiskPressure, PIDPressure, and allocatable capacity. Investigate NotReady nodes and verify node repair and replacement. Application probes cannot repair an unhealthy node.

Spread replicas across hosts and available zones with topology spread constraints or appropriate anti-affinity. Strict constraints can leave Pods pending if capacity is missing. Test placement and failure together. Node autoscaling adds compute; HPA adds Pod replicas. Neither supplies instant capacity.

Handle SIGTERM, stop accepting new work, and allow a measured termination grace period for in-flight requests. Workers must stop claiming jobs and acknowledge only durable completed results. Interrupted tasks must be safe to retry.

Example: probe configuration

# Deployment pod-template fragment; implement these endpoints.
# Tune the example timings and resources through load tests.
containers:
  - name: api
    ports: [{ containerPort: 8080 }]
    startupProbe:
      httpGet: { path: /health/startup, port: 8080 }
      periodSeconds: 5
      failureThreshold: 30
    readinessProbe:
      httpGet: { path: /health/ready, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    livenessProbe:
      httpGet: { path: /health/live, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    resources:
      requests: { cpu: 250m, memory: 256Mi }
      limits: { cpu: "1", memory: 512Mi }

Example: voluntary disruption budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ai-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: ai-api

Use matching Pod labels and at least three healthy replicas for this illustrative budget to permit one eviction. Adjust to your availability target and capacity.

04 / Control the work before it controls you.

Bound API concurrency and worker jobs. Admission limits protect your service; a shared provider limit protects quotas across every replica. More Pods must not multiply calls beyond your allowance. Return useful 429 responses and honor Retry-After guidance. Return a task ID only after asynchronous work is durably accepted.

Use a durable queue with retention, maximum task age, bounded retries, and a dead-letter path. Scale workers using queue age and depth where supported, while capping aggregate provider concurrency. CPU alone may miss workers waiting on remote calls. Custom metrics require an adapter; event-driven scaling requires an installed controller.

Retry transient failures using exponential backoff, jitter, attempt limits, and an overall deadline. Give retries one clear owner so SDK, worker, and queue layers do not multiply attempts. Use idempotency keys and durable deduplication so redelivery cannot repeat charges or other side effects.

Give every dependency connection and response deadlines. Circuit breakers stop repeated calls to failing services; separate concurrency pools contain impact. Evaluate fallback models for quality, data handling, and tool behavior before enabling them. Otherwise expose a clear delayed or failed task state.

Keep credentials out of Git, images, Helm values, rendered logs, and release history. Reference Secrets populated through a controlled process or installed external secret integration. Secret encoding is not encryption: restrict access and verify encryption at rest. Prefer short-lived workload identity where supported and rehearse rotation.

Use non-root containers and restricted permissions. NetworkPolicy must allow required DNS and dependencies and requires compatible networking for enforcement. Standard NetworkPolicy does not offer general hostname-based egress filtering; provider access may need an egress gateway.

05 / Make deployment easy. Make recovery deliberate.

Build once, test, scan, push an immutable image, package the chart, deploy to staging, run an end-to-end task, and promote reviewed artifacts. Serialize deployments to an environment. Use scoped short-lived access and verify the cluster context.

Waiting for resource readiness does not prove business success. Run a smoke test that submits a task, observes worker execution, and confirms the durable result. A failing post-release Helm test requires an explicit pipeline recovery decision; it does not automatically activate the preceding upgrade’s rollback flag.

Record the known-good chart, image digest, and configuration. For Helm-managed releases, inspect helm history and roll back to the selected good revision. For GitOps, revert desired state through the controller workflow so reconciliation does not reapply the faulty version.

A manifest rollback does not undo a database migration, restore deleted data, retract an external action, or replay lost queue messages. Use expand-and-contract schema changes, tested backups, and idempotent replay. Adjacent app versions must remain compatible during rolling updates.

In isolated staging, drain a node, interrupt a worker, simulate provider throttling, rotate a secret, deploy an unhealthy version, and restore a backup. Measure recovery time and inspect duplicate or lost tasks. Document evidence, operator actions, and limitations in the runbook.

Illustrative Helm 4 deployment

helm lint ./charts/ai-service -f environments/staging.yaml
helm template ai-service ./charts/ai-service -f environments/staging.yaml
helm upgrade --install ai-service ./charts/ai-service \
  --namespace ai-staging --create-namespace \
  -f environments/staging.yaml \
  --wait --timeout 10m --rollback-on-failure
helm test ai-service --namespace ai-staging --logs

These paths describe an illustrative chart, not supplied deployment artifacts. Helm 4 uses --rollback-on-failure; Helm 3 uses --atomic for comparable upgrade behavior. Pin versions and verify flags. A successful Helm release still needs application-level verification.

06 / The launch checklist.

• A clean environment can deploy the reviewed chart and immutable image.
• Startup, readiness, and liveness probes behave correctly under failure.
• Replicas span failure domains with capacity for rollout surge.
• Node drain respects the PDB; node failure leaves enough service capacity.
• Concurrency, provider rate limits, queue age, and retries stay bounded under load.
• Interrupted and duplicate jobs recover without repeated side effects.
• Secrets rotate without exposure or unexpected downtime.
• Application rollback and separate data restore have both been rehearsed.
• Customer-impact alerts have an owner and an escalation path.
• Recovery time and acceptable data loss are agreed and tested.

Technical references

• Kubernetes probes
• Disruptions and PodDisruptionBudgets
• Nodes and node health
• Horizontal Pod autoscaling
• Secret security practices
• Helm upgrade reference