DevOps / SRE · Playbook
DevOps / SRE Playbook
A practical guide to shipping and running software without heroics. Metrics describe health. Platforms describe the path. Incidents describe learning. Keep those ideas joined so you do not build a beautiful pipeline that nobody can operate, or a war-room culture that burns people out.
MAP-01 Reliability maps
Reliability & delivery maps
Pick the map that matches the decision in front of you.
DevOps and SRE mix delivery performance, reliability science, platform product thinking and security hygiene. Confusing them creates either endless tooling or a hero culture. Every framework below answers a different question.
Maps you will actually use
| Map | Answers | Shape |
|---|---|---|
| DORA metrics | Is delivery getting healthier? | Deploy frequency · lead time · change fail · recovery |
| SLO / error budgets | How reliable must we be, and what do we trade when we burn? | SLI · SLO · budget · policy |
| Continuous Delivery | Can we release safely on demand? | Trunk · pipeline · small batches · rollback |
| Observability | Can we ask new questions of a live system? | Metrics · logs · traces · continuous profiling |
| Incident management | How do we restore service and learn? | Command · comms · timeline · review |
| Secure SDLC / supply chain | Are we shipping known risk? | SAST · SCA · secrets · provenance · least privilege |
Choose in practice
- Exec debate on speed vs stability: DORA trends plus SLO burn, not vibes.
- Feature pressure eating reliability: write an error budget policy with Product Owner.
- Painful releases: shrink batch size and automate rollback before adding more CAB theatre.
- Pager storms: fix alert quality and SLOs before hiring more on-call bodies.
- Government service: join operability evidence to Service Standard and NCSC good practice continuously.
Titles vary: DevOps Engineer, SRE, Platform Engineer, Infrastructure Engineer. The accountability does not. Someone must make shipping routine, make failure visible, and reduce toil so product engineering can own more of the path. If that person is "whoever is on Slack at midnight", you have a reliability debt.
Accountability ladder (typical digital path)
- Associate / Junior: pipeline and observability craft on a contained service with coaching.
- DevOps / SRE: accountable for a service or platform slice: SLOs, CD, incidents, toil.
- Senior / Lead: multi-service reliability, coaches others, holds platform product coherence.
- Head of Platform / SRE: owns the profession frame, reliability strategy and hiring bar.
What you stop and start doing as you grow
You stop doing
- Being the only person who can release
- Measuring your week by tickets closed or fires fought
- Saying yes to every custom snowflake environment
You start doing
- Product-managing the platform: golden paths over tickets
- Publishing SLO and error budget policy with product
- Measuring your week by toil removed and MTTR trends
Three questions to run every day
If any answer is "no" for more than a week, that becomes the priority over the next exciting tool rewrite.
SLIs measure what users feel. SLOs set the target. Error budgets turn missed targets into an engineering policy, not a blame session. Without a policy, SLOs are posters.
Weak versus strong reliability signals
| Signal | Weak version | Strong version |
|---|---|---|
| Uptime claim | "We're five nines" | Documented SLO on a user journey SLI with measurement method |
| Alert | CPU high | Error budget burn or journey failure rate above threshold |
| Release decision | "Ship anyway, we'll watch" | Budget policy: freeze / slow / proceed with named residual |
| Success | No pages this week | Users succeeding; budget healthy; toil not hidden in silence |
SLO hygiene
- Choose SLIs from user journeys, not from every internal metric that is easy to graph.
- Set SLOs with Product Owner and SRO for public services; reliability is a product choice.
- Write the error budget policy: what happens at 50%, 25%, 0% remaining.
- Review burn in the same forums as roadmap trade-offs, not only in ops stand-ups.
DORA's four keys tell you whether flow and stability are improving together. Continuous Delivery is how you earn the right to go faster: every commit a release candidate, automated verification, fast rollback.
DORA in practice
| Metric | Use it to ask | Common distortion |
|---|---|---|
| Deployment frequency | Are batches getting smaller? | Deploying empty changes to game the number |
| Lead time for changes | Where is the queue? | Measuring only build time, ignoring wait for approval |
| Change fail rate | Are we breaking users? | Hiding failures by not calling them incidents |
| Failed deployment recovery | Can we restore quickly? | Manual heroics counted as a fine process |
CD minimum bar
- Trunk-based or short-lived branches; long-lived release branches are a smell.
- Pipeline is the path to production; manual snowflake deploys are exceptions with expiry.
- Automated tests and security gates with clear owners for overrides.
- Rollback or forward-fix path rehearsed, not theoretical.
- Feature flags where progressive delivery reduces blast radius.
Report DORA next to RAG with the Delivery Manager. Use trends to justify platform investment and to challenge scope pressure that worsens change fail rate.
Observability is the ability to understand system behaviour from outputs. Dashboards that only show known unknowns are not enough. Alerts should wake humans for user-harming conditions they can act on.
Signal design
Invest in
- Golden signals and journey SLIs
- Structured logs with correlation IDs
- Traces across service boundaries
- Runbooks linked from alerts
Retire
- Pages on every CPU blip
- Alerts nobody has actioned in 90 days
- Dashboards without owners
- "Watch the graphs" as a release plan
- Every pageable alert has a runbook, an owner and a link to the SLO or user impact.
- Delete or downgrade noisy alerts weekly until on-call is survivable.
- Instrument new features in Definition of Done: if it can break users, it can emit signals.
- Separate security detections from reliability pages with clear routing.
Name: …
User impact: …
SLI / threshold: …
First action: … · Runbook: …
When to silence / escalate: …
Incidents are inevitable in complex systems. Chaos comes from unclear roles, missing timelines and reviews that blame individuals without changing the system.
Incident roles
| Role | Owns | Does not own |
|---|---|---|
| Incident commander | Coordination, decisions, tempo | Deep-diving every technical fix alone |
| Technical lead | Diagnosis and remediation path | External stakeholder narrative |
| Comms lead | Status to users / SRO / support | Silent freelancing of technical changes |
| Scribe | Timeline and actions log | Optional nice-to-have in serious incidents |
Learning loop
- Blameless review within a short timebox while memory is fresh.
- Actions are system changes with owners and dates, not "be more careful".
- Feed learnings into SLO, pipeline, runbooks and toil backlog.
- Track reopen rates: if the same class returns, the review failed.
In government, align major incident comms with service support and SRO expectations early. Users and ministers hearing different stories is an incident of its own.
Toil is manual, repetitive, automatable work that scales with service growth. Platform engineering product-manages the internal developer experience so the happy path is secure, observable and fast.
Toil triage
| Pattern | First move |
|---|---|
| Same manual release steps weekly | Automate the path; forbid snowflake as default |
| Ticket for every environment | Self-service with guardrails |
| On-call doing known repetitive fixes | Automate remediation or remove the failure mode |
| Every team reinvents CI | Golden path templates with escape hatches |
- Measure toil hours. What you do not measure becomes badge-of-honour busywork.
- Treat platform work as a product: users are developers; roadmap is golden paths.
- Prefer paved roads over mandates without tooling.
- Escape hatches exist, but they are exceptions with owners, not the culture.
Shift security into the path of delivery: identity, secrets, dependency scanning, infrastructure policy and provenance. Partner with GRC on control intent; you own operable implementation.
Pipeline control patterns
| Control | Good looks like | Smell |
|---|---|---|
| SAST / lint | Fast, high-signal, owned baselines | Thousands of ignored findings |
| Dependency / SCA | Block criticals with fix path | Warn forever, never upgrade |
| Secrets detection | Pre-commit + pipeline + rotation playbook | "We'll rotate later" |
| IaC policy | Deny public exposure by default | Manual review of every plan as the only control |
| Provenance | Signed artefacts, known builders | Unsigned binaries from laptops |
- Overrides have named owners and expiry; log them where GRC can find them.
- Least privilege for deploy identities; no shared long-lived god credentials.
- Separate build and deploy trust; compromise of a laptop should not mean silent prod write.
- In government, map evidence to CAF / departmental security expectations continuously.
The facts do not change between the incident channel and the SRO update. The altitude, the framing and the ask do. Reliability work dies when engineers only speak graphs and sponsors only hear "it's fine".
Audience calibration
| Audience | Detail level | What they want |
|---|---|---|
| On-call peers | High, technical, linked runbooks | Clear next action and ownership |
| Product / DM | Medium, user impact and budget burn | Trade-offs and release implications |
| SRO / exec | Low, impact, ETA, residual risk | Honest status and a clear ask |
| Users / support | Plain language, what works now | What to do, when next update comes |
Impact: who / what is broken
Status: investigating / mitigating / monitoring
Next update by: [time]
Ask: specific help, or "no action for you"
SLO: … · Budget remaining: …
Burn driver: …
Options: proceed / slow / freeze reliability work first
Recommendation: … · Decision by: …
Words to drop
| Drop | Use |
|---|---|
| "It's flaky" | "Failure rate X on journey Y since Z; mitigating by…" |
| "Just a blip" | "Impact window …; residual risk …; follow-up …" |
| "We need a war room" | "Incident declared; commander is …; channel is …" |
Operability belongs in Definition of Done with Product Owner and testers. Date pressure that skips instrumentation, rollback and access hygiene borrows reliability from the future.
Operability checklist
- SLIs or dashboards updated for new user-facing paths.
- Alerts reviewed; no orphan pages introduced.
- Rollback or mitigate path documented and tested in kind.
- Secrets, roles and network exposure reviewed.
- Support / runbook notes exist when users will feel the change.
- Performance and accessibility budgets respected for public services.
For UK public services, join performance, security and reliability evidence to Service Standard continuously. Assessment should not be the first load test or the first access review.
Under pressure, the instinct is to add more alerts, more approvals or a war room. Diagnose first. Pager storms and war-room releases are usually symptoms of weak SLOs, fat batches or missing automation.
| Symptom | Likely cause | First move |
|---|---|---|
| Pager storm | Alerts not tied to user impact; flapping thresholds | Silence non-actionable pages today; rewrite top noisy alerts against SLOs this week |
| War-room every release | Batches too large; weak rollback; untested pipeline | Shrink next release; rehearse rollback; forbid manual prod steps as default |
| DORA looks fine, users angry | Gaming metrics or wrong SLIs | Re-anchor SLIs on journeys; stop empty deploy theatre |
| Error budget ignored | No policy with product | Write burn policy with PO/SRO; apply at next burn threshold |
| Only one person can release | Snowflake knowledge; no golden path | Pair and document; automate path; make specialist absence a drill |
| Toil dominates sprint | No toil budget; platform is a ticket queue | Cap interrupt; schedule automation for top toil source |
| Security gates always overridden | Noise or impossible policy | Tune signal; timebox overrides; fix the false positive class |
| Incidents repeat | Reviews without system actions | Reopen last review; require owned system change before close |
| Observability gaps mid-incident | Instrumentation not in DoD | Add missing signals as P1 follow-up; update DoD this fortnight |
| CAB theatre blocking CD | Change process built for rare releases | Standard changes via pipeline evidence; reserve humans for high risk |
The one-minute checklist
- Name user impact in one sentence.
- Is the pain alert quality, batch size, SLO policy, toil or security gate design?
- Which decision restores service or unblocks the most, and who owns it?
- Tell the SRO / product lead before they hear a distorted version.
- Protect on-call from noise while you sort the system fix.
Glossary anchors
- SLI / SLO: indicator / objective for reliability
- Error budget: allowed unreliability under the SLO
- DORA: four key delivery performance metrics
- CD: Continuous Delivery
- MTTR: mean time to restore (use carefully; prefer distribution)
- Toil: manual, repetitive, automatable ops work
- Golden path: supported default way to build and ship
- Blameless review: learn from systems, not villains
- CAF: NCSC Cyber Assessment Framework
- SBOM: Software Bill of Materials
- RAG: Red, Amber, Green status
- SRO: Senior Responsible Owner
Companion: the cheatsheet distils every section into a one-screen field reference.