Taking an OCR platform from 28% job failures to 0.18%

// 01 the problem

The OCR platform turns scanned documents into structured data for downstream pipelines: NVIDIA Triton with 34 models on GPU nodes, fronted by API and worker services. Jobs were silently expiring. 28% of OCR runs failed on the development cluster, a backlog of 81 jobs sat unprocessed, and nobody could say why, because the workers were not crashing. Separately, a data team needed a dedicated OCR cluster for a large batch job, fast, on two GPU machines that had never run the stack.

// 02 architecture

Clients hit a first HAProxy tier that balances across three identical OCR workloads sharing one Redis queue. Each worker calls Triton through a second, per-node HAProxy tier that prefers the Triton on its own machine. Prometheus and Loki collect metrics and logs; Grafana carries the Triton board; k6 breakpoint tests run against each release.

// 03 what I did and why

1. Found the silent failure in the scheduler, not the code. Worker pods had no CPU requests, so Kubernetes classed them BestEffort and starved them under contention (cgroup weight 1). Setting requests (3 CPU, 12 GiB) moved them to Burstable; expiries stopped, failures fell from 28% to 0.18%, the backlog drained to zero. The fix is in Git, not only on the cluster.
2. Designed the dedicated cluster to look like production. Two HAProxy tiers, three workloads, one Redis: the same shape as prod, so runbooks and dashboards transfer. A real OCR job completed in 4 seconds end to end on the new cluster.
3. Measured the network before trusting it. The first design let workers call Triton across machines; both had 100 Mb/s ports, the cross-node traffic saturated them and packet loss hit 43–52%. Pinning each worker to its local Triton brought loss to 0% and inbound traffic from 12 MB/s to 42 KB/s. The cross-node design was removed the same day.
4. Made the model repository versioned. Triton's flat store became a versioned repository with a rollback copy; 13 obsolete models were removed after checking no caller referenced them; the prod repository was mirrored to dev (5.8 GiB, 186 objects, 34/34 models READY).
5. Protected user data on start-up. An init container verifies the data volume is mounted before the service starts; the request-file TTL went from 24 hours to 1 hour.
6. Traced requests end to end. Trace IDs now flow from logs to OpenTelemetry traces (3 application commits, 17 new tests).
7. Recovered a 23-hour GPU outage on a dev node: a kernel upgrade shipped without the matching driver module. Installing it and resetting the control pod restored the GPU in about 20 seconds; the kernel package is now held and a pre-reboot checklist exists.

// 04 measured results