Anurag’s Read

Slack AI: The Path to Multi-Cloud.

A four-stage story of how Slack AI moved from self-run SageMaker endpoints to a multi-cloud routing layer, with the reason for each move. It covers capacity planning, safe migration, circuit breakers and the real costs of using several clouds.
Score
8.5
Verdict
Must read
Blog and date
Slack,
Original length and level
17 min, advanced
Topics
llm serving, multi-cloud, reliability, migrations
Read the original

Context

Slack AI served LLM features such as channel summaries and Recap from AWS for about three years. The team wanted resilience against regional outages, tolerance of GPU scarcity, and access to the newest models. Each serving stack fixed some limits and exposed others.

Early on, engineering time went into plumbing and capacity planning. Later, a single cloud still left Slack exposed to provider-wide outages and vendor-exclusive models.

Architecture

Slack AI moved through four phases. It began on SageMaker across several AWS regions, with an escrow VPC. It then moved to Bedrock Provisioned Throughput, and then to a hybrid of Provisioned Throughput and On-Demand.

In early 2026 it added GCP Vertex AI. An Intelligent Routing Layer now picks a model for each feature using internal quality benchmarks. Each feature has a backup model, and experiment rules route a percentage of traffic to test models.

A circuit breaker watches endpoint health, and an API Normalization layer maps provider errors to one handling path. A unified telemetry stack covers all clouds.

Mechanisms

M1

SageMaker phase

Slack ran model containers on SageMaker in several regions. An escrow VPC kept Slack data private and the weights of the model provider unreachable. The team built cross-region IAM roles, endpoint load balancing, capacity planning, and auto-scaling. Scaling was slow because of initialization time, and A100 and H100 instances were scarce.

The team over-provisioned to meet peak SLAs. In early 2024 they added On-Demand Capacity Reservations and cron-based scaling.

M2

Bedrock migration

Provisioned Throughput capacity is bought in Model Units, and each unit gives a fixed tokens-per-minute rate. Slack moved Provisioned Throughput first, for latency-sensitive features, and On-Demand later, for scheduled batch work. The team got legal, security, and FedRamp sign-off. Load tests sized the units to the SageMaker baseline, and A/B and evaluation comparisons checked quality and latency.

Then shadow requests ran, and feature flags shifted traffic gradually with near-instant rollback. The team reports no customer-facing incidents and forecasting several weeks ahead.

M3

Hybrid routing and spillover

Provisioned Throughput served high-volume, latency-sensitive features. On-Demand served bursty asynchronous work. A spillover pattern sent requests beyond reserved limits to On-Demand endpoints. Cross-region routing used US regions inside data boundaries.

Each feature had a model hierarchy for automatic fallback. Fallback signals were higher time to first token, throttling errors, and falling customer feedback.

M4

Circuit breaker

The breaker watches time to first token, 5xx spikes, and p90 latency thresholds. When it trips, traffic goes to another endpoint. It then enters a partial-open state and lets a small trickle through. Traffic ramps back up while health holds, and then the breaker closes.

The post gives no threshold or ramp values.

M5

API normalization and authentication

The normalization layer maps provider errors and rate-limit signals onto one handling path. Examples are Rate Limit Exceeded and Throttling Exception. Authentication is secretless. This lets the router treat all providers the same.

The team also built a unified telemetry stack, because native cloud dashboards were not enough.

Design decisions

No.ChoseInstead ofWhy
D1Hybrid Provisioned Throughput plus On-Demand with spilloverAll Provisioned Throughput or all On-DemandAll Provisioned Throughput left idle capacity and locked in commitments. All On-Demand made uptime vary for latency-sensitive features.
D2Move from SageMaker to BedrockStaying on SageMakerEngineering time went to plumbing, and new Anthropic models reached SageMaker weeks or months after Bedrock.
D3Add GCP Vertex AIA single-cloud footprintA single cloud stayed exposed to provider-wide outages and vendor-exclusive models.
D4Migrate Provisioned Throughput firstIt reduced risk, and On-Demand followed.

Key numbers

  • Multi-cloud brought about 10% better quality on complex reasoning tasks.
  • Latency fell about 67% for high-velocity, low-token workloads.
  • Experiment feedback loops shrank from weeks to days.
  • Provisioned Throughput commitments last one to six months.
  • The stack evolved over about three years in four phases.
  • The Bedrock migration reported zero customer-facing incidents.

Failures and limits

  • Provisioned Throughput sized for global peak left capacity idle off-peak and on weekends.
  • Commitments of one to six months slowed model upgrades.
  • On-Demand pools gave variable service and less control over regional capacity.
  • Multi-cloud makes cost attribution harder across billing systems and widens the on-call skill need.
  • The post gives no circuit-breaker thresholds, cost figures, or uptime numbers.
  • It reports no specific incidents.

Lessons you can use

  • Put an abstraction layer between features and providers before you add a second provider.
  • Name a backup model for every feature.
  • Count slow responses, such as p90 spikes, as soft failures.
  • Plan legal and compliance work in parallel with engineering.
  • Treat the architecture as a living document.

Glossary

Provisioned ThroughputA Bedrock plan that reserves model capacity for a fixed term.
Model UnitA block of reserved capacity with a fixed tokens-per-minute rate.
Time to first tokenThe delay before a model returns the first piece of its answer.
Circuit breakerA guard that stops traffic to a failing endpoint and tests it again later.
Shadow requestA copy of live traffic sent to a new system whose answer users do not see.