Anurag’s Read

Deploy Safety: Reducing customer impact from change.

An honest program-level account of cutting change-caused incidents, including metric choice, trailing data, adoption problems and the gain from automatic rollback. It is useful for anyone who owns reliability across many teams.
Score
8.6
Verdict
Must read
Blog and date
Slack,
Original length and level
11 min, intermediate
Topics
reliability, deployments, incident reduction, engineering management
Read the original

Context

Slack's own changes, mostly code deploys, triggered 73% of customer-facing incidents. The fleet had hundreds of services and many deployment systems. Customers said interruptions felt disruptive after about 10 minutes.

They expected less tolerance as Agentforce launched in 2025.

Architecture

This post describes a program and not one system. The team set four goals. They wanted automatic detection and remediation within 10 minutes and manual remediation within 20 minutes.

They wanted to catch bad deployments before 10% of the fleet, and they kept development velocity. The webapp backend caused the most change-triggered incidents, so it served as the template for other services.

Mechanisms

M1

Success metric

The top metric is hours of customer impact from high-severity and selected medium-severity incidents that a change triggered. Severity reflects the impact at the time of rating. So the team filters medium incidents by a customer-impact threshold, which the post does not define. The metric uses trailing incident data, so the full effect of a project appears after 3 to 6 months.

M2

Investment selection

The team first invested broadly, then scaled what worked and cut the least effective areas. Each project had to improve one of four effects. These were earlier detection, faster automatic remediation, faster manual remediation, and a smaller blast radius through isolation boundaries. Automation gave better results than faster manual remediation.

M3

Webapp backend sequence

The team built automated metric monitoring first, which took about one quarter. Next came alert-based and manual rollback, to confirm that alerts matched customer impact. Then the team added automatic deployments and rollback, and a manual frontend rollback. A central deployment orchestration system, inspired by ReleaseBot and AWS Pipelines, extends this beyond Kubernetes-based deployments.

Design decisions

No.ChoseInstead ofWhy
D1Add automatic rollbackFaster manual remediation aloneManual changes alone were not enough. Results improved sharply after automatic rollbacks began in April 2024.
D2Use one consistent metricA long debate about a perfect metricThe metric is an imperfect proxy for customer sentiment, but a stable measure lets teams track progress.
D3Start from the webapp backendIt caused the most change-triggered incidents, so its lessons could extend to other services.

Key numbers

  • 73% of customer-facing incidents came from Slack's own changes.
  • Goals were 10 minutes for automatic and 20 minutes for manual remediation.
  • Customer impact hours fell about 90% from the peak by January 2025.
  • The peak ran from February to April 2024.
  • Full impact of a project takes 3 to 6 months to appear.
  • Executives reviewed progress every 4 to 6 weeks.

Failures and limits

  • Some projects had less impact than expected, and the team treated this as input for investment choices.
  • The main metric lags and is an imperfect proxy for customer sentiment.
  • Engineers adopted unfamiliar rollback tools slowly.
  • Communication with general engineers lagged behind alignment with management.

Lessons you can use

  • Pick one consistent metric and accept its flaws.
  • Plan for lagging data by adding interim checks, such as detection performance.
  • Prefer automation over faster manual steps for remediation.
  • Contact teams directly, because their pain points differ.

Glossary

Blast radiusThe share of users or systems that a bad change can affect.
Trailing metricA measure built from past events, so it shows results late.
RollbackA return of a system to the previous known-good version.