Zero-Downtime Deployments and Cloud Cost Control: A DevOps Playbook

What zero downtime actually requires

Most production incidents begin with a change: a deploy, a configuration edit, a migration or a feature switched on. That is not an argument for changing less. Teams that release rarely bundle more risk into each release and are slower to recover when it goes wrong. The goal is to make change cheap and boring, so that any single deploy carries little risk and can be undone in seconds.

In practice, zero downtime rests on four properties. New and old versions of the application must be able to run side by side, because during any real rollout they will. The database schema must work with both versions. Traffic must only reach instances that are genuinely ready to serve it. And something other than a tired engineer must be watching the health of the release and able to reverse it automatically.

None of this requires a particular platform. Kubernetes, a managed container service and a pair of virtual machines behind a load balancer can all deliver it. What matters is that the four properties are designed in, tested and rehearsed, rather than assumed because the deployment tool's documentation says rolling updates are supported. A useful test is a game day: deploy a deliberately broken build to staging under synthetic load and time how long it takes, without human help, for traffic to return to the healthy version.

Blue-green, rolling and canary: choosing a release strategy

A rolling update replaces instances a few at a time and is the default in Kubernetes. It needs no extra capacity beyond a small surge, which makes it the right starting point for most stateless services. Its weakness is that a bad version reaches every instance before you have much signal, and rolling back is another rolling update. Set maxUnavailable to zero for user-facing services, and make sure pods drain cleanly on termination by handling SIGTERM and giving in-flight requests time to finish.

Blue-green runs a complete second environment, switches traffic to it in one step, and keeps the old one warm for instant rollback. It suits releases that must change atomically and systems where rollback speed matters more than infrastructure cost, because you pay for double capacity during the cutover. Canary sends a small slice of real traffic to the new version, compares its error rate and latency against the stable version, and widens the slice only while the numbers hold. Tools such as Argo Rollouts and Flagger automate the analysis. Canaries need enough traffic to produce a meaningful signal, so they suit busy services better than internal tools with a handful of users.

Feature flags complete the picture by separating deploying code from releasing behaviour. Ship the code dark, turn it on for staff, then a percentage of users, and keep a kill switch that does not need a deploy. Treat flags as inventory with owners and expiry dates, because a codebase with hundreds of stale flags becomes its own reliability risk. Remove a flag within a sprint or two of reaching full rollout, and test the off path as seriously as the on path.

• Use rolling updates for stateless services that have good health checks and moderate traffic.
• Use blue-green for atomic cutovers, major runtime upgrades and services where rollback must be instant.
• Use canaries with automated metric analysis for high-traffic, user-facing services where a bad release is expensive.
• Use feature flags whenever business timing, not engineering readiness, should decide when users see a change.

Backwards-compatible database migrations with expand and contract

The database is where zero-downtime plans usually fail, because every release strategy above assumes old and new code can share one schema. A migration that renames a column or adds a non-nullable field without a default breaks the old version the moment it runs, and the old version is still serving traffic. The answer is the expand and contract pattern: every schema change is split into additive steps that are safe for both versions, followed much later by a cleanup step.

Operational details matter as much as the pattern. On PostgreSQL, build indexes with CREATE INDEX CONCURRENTLY, set a short lock_timeout on migration sessions so a blocked migration fails fast instead of queueing every query behind it, and backfill large tables in small batches rather than one long transaction. Run migrations as a separate, observable step in the pipeline, not as a side effect of application start-up on every pod.

• Expand: add the new column, table or index in a way the current code ignores, such as a nullable column.
• Dual write: release code that writes to both old and new structures while still reading from the old one.
• Backfill: copy historical data in batches, with progress tracking and the ability to pause.
• Switch reads: release code that reads from the new structure, verified against the old one for a period.
• Contract: stop writing the old structure and drop it in a later release, once rollback past this point is no longer needed.

Health checks, SLOs and automatic rollback

A readiness check should answer one question: can this instance serve a real request right now? It should verify that configuration loaded and critical dependencies are reachable, and it must be cheap. A liveness check is different and narrower, detecting a process that is stuck, and it should never depend on downstream services, or one slow database will restart your whole fleet. Get these two wrong and the orchestrator will confidently route traffic to broken pods or kill healthy ones.

Service level objectives turn health into a number the business agrees with. An availability target of 99.9 percent over thirty days allows roughly 43 minutes of failure; 99.95 percent allows about 22. The gap between the target and perfection is the error budget. When releases burn it quickly, the team slows down and invests in reliability; when plenty remains, the team can ship more aggressively. Alert on how fast the budget is burning, across a short and a long window, rather than on every CPU spike.

Automatic rollback ties it together. The deployment controller watches the same SLO metrics during a rollout and reverts if error rate or latency for the new version crosses a threshold. That only works with good telemetry: request rate, errors, latency and saturation per service and per version, traces through OpenTelemetry, and dashboards in Prometheus and Grafana that someone has actually looked at before the incident.

The cost levers that actually move the bill

Most cloud waste comes from a few predictable places, and they are worth tackling in order of effort. Start with right-sizing: compare requested CPU and memory against observed usage over a few weeks, and shrink instances or Kubernetes requests that sit mostly idle. Then make capacity follow demand with horizontal autoscaling for services and cluster autoscaling for nodes, and switch off development and staging environments outside working hours, which by itself removes roughly two-thirds of their weekly running time.

Only once usage is right-sized should you commit to it. AWS Savings Plans and Reserved Instances, Google Cloud committed use discounts and Azure reservations all trade a one or three year commitment for a lower rate, and committing to an oversized baseline locks in the waste. Spot and preemptible capacity is far cheaper but can be reclaimed with little warning, two minutes on AWS, so it suits batch jobs, CI runners, data processing and stateless workers that checkpoint, not your primary database.

Storage and data transfer are where bills surprise people. Move ageing objects to infrequent-access and archive tiers with lifecycle rules, but check minimum storage durations and retrieval fees first. Watch for NAT gateway processing charges, cross-zone traffic between chatty services, internet egress from serving large files without a CDN, and log ingestion volumes that quietly exceed the cost of the servers producing them.

• Right-size first, autoscale second and commit third, never the other way round.
• Schedule non-production environments to stop at night and at weekends.
• Run interruptible workloads such as CI, batch and data jobs on spot capacity with checkpointing.
• Add lifecycle rules to every bucket and retention limits to every log group.
• Route object storage traffic through gateway endpoints and serve static assets through a CDN to cut NAT and egress charges.

Tagging, per-team visibility and a monthly FinOps review

Nobody optimises a cost they cannot see or do not own. Every resource should carry a small, enforced set of tags: owner, team, service, environment and cost centre. Enforce them in Terraform modules and policy checks in the pipeline, so untagged resources fail review rather than appearing as an unexplained line on next month's invoice. For Kubernetes, where many teams share nodes, use a cost allocation tool such as OpenCost to split cluster spend by namespace and label.

With ownership in place, cost becomes a routine conversation rather than a quarterly panic. Publish a per-team view of spend alongside a unit metric that matters to the business, such as cost per thousand orders, per active customer or per million API calls. Absolute spend should grow with the business; cost per unit should not. Set budget alerts and anomaly detection so a runaway job is noticed within a day, not at month end.

• Review the top ten cost changes from last month and assign an owner to each one.
• Check commitment coverage and utilisation against current right-sized usage.
• List untagged and orphaned resources, such as unattached volumes, old snapshots and idle load balancers, and delete or claim them.
• Track unit cost per team against the previous three months and agree one improvement per team.
• Record decisions in a short log so next month's review starts from facts, not memory.

The right-sized option for small teams

Not every product needs Kubernetes, multiple regions and a service mesh on day one. For an early-stage product with one team and modest traffic, a single well-run virtual server running containers behind a reverse proxy, paired with a managed PostgreSQL service, object storage and a CDN, is often the most reliable and cheapest option available. The managed database handles backups, point-in-time recovery and patching, which are the parts small teams most often get wrong.

Zero-downtime principles still apply at this scale. Run the new container alongside the old one, let the proxy health-check it, switch traffic and stop the old container; tools such as Kamal package exactly this flow. Keep the server defined in code, ship logs and metrics off the box, and rehearse rebuilding it from scratch. Graduate to an orchestrator when several teams need to deploy independently, when one machine can no longer absorb your peaks, or when your availability target requires redundancy across zones.

How we would approach it

We usually begin with a two-part audit: a walk through the release path from merge to production, looking for the four zero-downtime properties, and a cost breakdown by owner, service and environment. Fixes are then prioritised by risk and payback, typically health checks and migration discipline first, then right-sizing and tagging, then canaries and commitments. Each change ships with a before and after measurement, such as deploy duration, failed-change rate or monthly spend per service, so the improvement is visible rather than asserted.

This work sits with our DevOps & SRE Engineering and Cloud Solutions & Infrastructure teams, and it maps to the Zero-Downtime Cloud Deployment step of our delivery process. If you want a rough sense of effort before talking to anyone, the Cloud & DevOps option in our Budget Calculator gives an indicative range; a short scoping call then turns it into a concrete plan.