Introduction

02:14 — p99 latency on checkout-api triples. 02:15 — a Pulse investigation starts. This is a walkthrough of what happened in between, reconstructed from the investigation’s own artifacts.

The investigation, step by step

Collectors pulled the four signals that matter in the first minute: metrics around the anomaly window, logs from the affected pods, Kubernetes events, and the deployment timeline. One fact stood out — a deploy had landed at 02:02.

Hypothesis agents tested three candidates: the new deploy, a memory leak, and an upstream dependency. The evidence converged fast — the new revision’s pods were hitting memory limits and OOM-restarting under load, something the pre-deploy traffic level had masked.

The report

Root cause: memory limit too low for the new revision’s cache behavior, 91% confidence. Contributing factor: a traffic bump from a scheduled job. Suggested fixes: raise the limit (low risk) or roll back (low risk, immediate). Slack summary: four sentences.

The on-call engineer rolled back straight from the summary without opening a dashboard. Total time from anomaly to resolution: eleven minutes, most of it human decision time.

The takeaway

Nothing in this story is heroic — that’s the point. Investigations are mechanical work at machine speed, so the human only does the judgment.