Rollbacks

The plain-language idea

The instinctive definition of "roll back a bad deploy" is: git revert the offending commit, push, let the pipeline rebuild and redeploy. That works, sometimes — but it's the slow and fragile version, and it's not actually what "rollback" means in a mature pipeline.

A real rollback redeploys the previous known-good artifact — the exact build that was already running successfully before the bad deploy, byte-for-byte, pulled back out of wherever it was stored (a registry, an artifact store) and put back in front of traffic. It does not run pnpm build again, does not run docker build again, does not touch source control at all. This distinction matters for a very practical reason: git revert + rebuild means running your entire build pipeline again, under pressure, during an incident — and a rebuild can fail (a dependency got yanked, a registry is down, a flaky test blocks CI) or, worse, can succeed while producing something subtly different from what was actually running before, because "same source" doesn't guarantee "same build output" (this is the same promotion problem from the environments lesson, showing up at the worst possible time). Redeploying a stored artifact sidesteps all of that — it's just moving a known-good file back into place, which is fast and doesn't depend on anything except your artifact store being up.

See it in a real thing: rolling back a Kubernetes deployment

# see what's actually deployed right now, and its rollout history
kubectl rollout history deployment/myapp
# REVISION  CHANGE-CAUSE
# 12        image=myapp:a3f9e21 (the bad deploy)
# 11        image=myapp:8b2c04d (last known-good)

# roll back to the previous revision — no rebuild, just re-point at the old image
kubectl rollout undo deployment/myapp

# or target a specific known-good revision explicitly
kubectl rollout undo deployment/myapp --to-revision=11

# confirm the rollback actually landed
kubectl rollout status deployment/myapp

kubectl rollout undo never touches source code or runs a build — it tells Kubernetes to redeploy the image tag from revision 11, the one already proven to work, using the same rolling-update mechanism a forward deploy uses. This is why tagging every build with an immutable identifier (a commit SHA, not latest) from the environments-and-promotion lesson isn't just good hygiene — it's the thing that makes this command possible at all. If every deploy overwrote the same latest tag, there'd be nothing distinct to roll back to.

v1 is running fine. v2 deploys — and the error rate spikes right after it goes live.

BackNext

What people usually miss

Watch out

• Database migrations don't roll back with the app. If the bad deploy ran a migration that dropped a column or changed a type, rolling the app back to old code doesn't undo the schema change — the old code now points at a schema it was never written against. This is why most teams write migrations forward-only and additive (add the new column, backfill, migrate reads, then drop the old column in a later, separate deploy) so any single app rollback stays safe regardless of migration state.
• "Redeploy the old artifact" only works if the old artifact still exists. Registries and artifact stores commonly have retention/cleanup policies — if last week's image already got garbage-collected, there's nothing to roll back to, and you're forced into a rebuild under pressure, exactly the situation you were trying to avoid.
• A rollback is not automatically the safe choice. If the bad deploy already wrote data in a new format, or a downstream consumer of an API has already started depending on the new response shape, rolling back can itself cause a second incident. Rollback is the default instinct, not an unconditional rule.

The genealogy: this is why deployment strategy choice matters upstream

The deployment-strategies lesson showed blue-green's rollback as "flip the router back" and canary's as "cut traffic back to zero" — both are the same underlying idea as kubectl rollout undo: reactivate a known-good artifact instead of rebuilding one. How fast and how cleanly you can roll back was actually decided back when you picked a deployment strategy, not at incident time.