Case study 02 · C-OpenAI (CMC Corporation) · Sep – Nov 2026
A digital workforce that runs infrastructure operations under human control
−30%
operator workload (estimate)
< 2 min
hang detection, 6/6 simulated events
13
specialised agents, one per component
100
ArgoCD apps Synced / Healthy
// 01 the problem
One DevOps function was responsible for about nine engineering teams, a 100-application ArgoCD estate, a GPU inference cluster and an OCR platform on Docker Swarm. Each component had its own repos, dashboards, failure modes and tribal knowledge. The work that filled the day was not design: reading logs, correlating alerts, writing the same GitOps changes across repos, producing status reports. Hiring was not an option on the timeline. The question was whether AI agents could carry that load without ever being the thing that broke production.
// 02 architecture
AI agents do the diagnosis; a human approves every change
↑ approve · push · deploy ↓
↓ read-only diagnostics · local commits · proposed GitOps changes
Each platform component gets its own agent with access to that component's repositories, cluster read access and runbooks. A manager agent assigns work and collects everything that needs a human decision. Agents produce diagnostics, local commits and proposed changes; nothing reaches a cluster or a protected branch without the engineer on duty reviewing and pushing it.
// 03 what I did and why
- One agent per component, not one agent for everything. A single assistant with access to everything is both a security risk and a context problem. Scoping each agent keeps its knowledge deep and its blast radius small; thirteen were running by November.
- A hard human gate. Agents may read anything and commit locally; only I push, merge to protected branches or touch a cluster. The rule came out of an early incident where a chained replica rollout caused a short error spike, and it has held since.
- A manager agent as the single intake. Teams ask one place; it routes to the right agent, deduplicates findings and keeps the approval queue visible.
- Standard outputs. Every agent writes the same daily report shape (what changed, evidence, open items), so the Teams report is assembled, not written.
- Automation where agents found repeated toil: a two-minute hang detector for GPU serving instances, a daily GPU-node health report, automated image-tag bumps across GitOps repos.
// 04 measured results
| Metric | Value | Source |
|---|---|---|
| Operator workload and troubleshooting effort | about 30% lower (estimate, not instrumented) | own tracking |
| Hang detection on GPU serving instances | under 2 minutes; 6/6 simulated events caught, no duplicates | detector log |
| Production incidents caught by the detector | 1 node outage, 2 GPU-node reboots | alerts log |
| Platform state after rollout of the model | 100 ArgoCD applications, all platform apps Synced / Healthy | ArgoCD, 5 Nov 2026 |
| Root causes found by agents, fixed by me | login-chain failure (RBAC seed), API 503s from missing storage config (191 → 0), spoofable X-Forwarded-For at ingress | session reports |
Lesson. The agents are only as safe as the gate in front of them. The incident that shaped the rules was not an agent hallucinating; it was a rollout sequence that was correct in isolation and wrong under load. The fix was procedural, one replica at a time with a health check before the next, and it is now in every agent's instructions.
// 05 links
Agent operating rules, the manager's report template and the hang-detector script can be published in sanitised form.