M1SageMaker phase
Slack ran model containers on SageMaker in several regions. An escrow VPC kept Slack data private and the weights of the model provider unreachable. The team built cross-region IAM roles, endpoint load balancing, capacity planning, and auto-scaling. Scaling was slow because of initialization time, and A100 and H100 instances were scarce.
The team over-provisioned to meet peak SLAs. In early 2024 they added On-Demand Capacity Reservations and cron-based scaling.
M2Bedrock migration
Provisioned Throughput capacity is bought in Model Units, and each unit gives a fixed tokens-per-minute rate. Slack moved Provisioned Throughput first, for latency-sensitive features, and On-Demand later, for scheduled batch work. The team got legal, security, and FedRamp sign-off. Load tests sized the units to the SageMaker baseline, and A/B and evaluation comparisons checked quality and latency.
Then shadow requests ran, and feature flags shifted traffic gradually with near-instant rollback. The team reports no customer-facing incidents and forecasting several weeks ahead.
M3Hybrid routing and spillover
Provisioned Throughput served high-volume, latency-sensitive features. On-Demand served bursty asynchronous work. A spillover pattern sent requests beyond reserved limits to On-Demand endpoints. Cross-region routing used US regions inside data boundaries.
Each feature had a model hierarchy for automatic fallback. Fallback signals were higher time to first token, throttling errors, and falling customer feedback.
M4Circuit breaker
The breaker watches time to first token, 5xx spikes, and p90 latency thresholds. When it trips, traffic goes to another endpoint. It then enters a partial-open state and lets a small trickle through. Traffic ramps back up while health holds, and then the breaker closes.
The post gives no threshold or ramp values.
M5API normalization and authentication
The normalization layer maps provider errors and rate-limit signals onto one handling path. Examples are Rate Limit Exceeded and Throttling Exception. Authentication is secretless. This lets the router treat all providers the same.
The team also built a unified telemetry stack, because native cloud dashboards were not enough.