Canary Deployment:

 

Canary Deployment: Safe Rollouts with Minimal Risk

📅 Published: July 2026
⏱️ Estimated Reading Time: 20 minutes
🏷️ Tags: Canary Deployment, Progressive Delivery, Kubernetes, Istio, DevOps

---

Introduction: What is Canary Deployment?

Canary deployment is a strategy where you roll out a new version to a small subset of users first, then gradually increase the percentage over time. If the new version shows problems, you can stop the rollout immediately with minimal impact.

The name comes from the "canary in a coal mine" – miners used to carry canaries to detect toxic gases. If the canary died, they knew to evacuate. Similarly, if the canary (new version) fails, you know to roll back before all users are affected.

Why Canary deployments matter:

• Minimize blast radius: Only a small percentage of users are affected if something goes wrong

• Real-world validation: Test with real user traffic, not just synthetic tests

• Gradual confidence: Increase traffic as confidence grows

• Fast rollback: Stop the rollout immediately if issues are detected

• A/B testing: Test new features with a subset of users

---

Part 1: How Canary Deployment Works

The Gradual Rollout Process

A Canary deployment proceeds in stages:

text

Stage 1: 0% → Deploy new version (no traffic yet)
Stage 2: 5% → Route 5% of users to canary
Stage 3: Monitor metrics (errors, latency, logs)
Stage 4: 20% → Increase to 20% if metrics are healthy
Stage 5: Monitor metrics again
Stage 6: 50% → Increase to 50%
Stage 7: Monitor metrics
Stage 8: 100% → All users on new version
Stage 9: Terminate old version

The Decision Points

At each stage, you check:

• Error rate: Are there more 5xx errors in the canary?

• Latency: Is the canary slower?

• Logs: Are there any new exceptions?

• Business metrics: Is the conversion rate affected?

• User feedback: Are users reporting issues?

If any metric crosses a threshold, you stop the rollout and roll back.

Rollback Procedure

If problems are detected:

1. Stop the canary rollout immediately

2. Route 100% of traffic back to the old version

3. Investigate the issue in the canary

4. Fix the issue and restart the rollout

Rollback is instant because the old version is still running.

---

Part 2: Canary vs Other Deployment Strategies

Strategy	Downtime	Rollback Speed	Risk	User Impact	Complexity
Canary	Zero	Immediate	Low	Gradual	High
Blue-Green	Zero	Immediate	Low	All at once	Medium
Rolling Update	Zero	Gradual	Medium	Gradual	Low
Recreate	Yes	Slow	High	All at once	Low
A/B Testing	Zero	Immediate	Low	Targeted	High

Canary vs Blue-Green:

Aspect	Canary	Blue-Green
Rollout speed	Gradual	Instant
Risk	Lower (small subset first)	Higher (all users at once)
Testing	Real traffic on canary	Testing on Green before switch
Infrastructure cost	Same (two versions running)	Same (two versions running)
A/B testing	Yes	No
Complexity	Higher	Medium

When to use Canary:

• You need to test with real user traffic

• You want to minimize blast radius

• You have complex applications with many dependencies

• You want to do A/B testing

• You can monitor metrics in real-time

When to use Blue-Green:

• You want simpler rollout mechanics

• You can test fully in a staging environment

• You need instant full rollout

• You don't need A/B testing

---

Part 3: Implementing Canary in Kubernetes

Using Deployment and Service

Step 1: Create both deployments

yaml

# stable-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-stable
  labels:
    app: myapp
    version: stable
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: stable
  template:
    metadata:
      labels:
        app: myapp
        version: stable
    spec:
      containers:
      - name: myapp
        image: myapp:1.0
        ports:
        - containerPort: 8080
---
# canary-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-canary
  labels:
    app: myapp
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      version: canary
  template:
    metadata:
      labels:
        app: myapp
        version: canary
    spec:
      containers:
      - name: myapp
        image: myapp:2.0
        ports:
        - containerPort: 8080

Step 2: Create the Service

yaml

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: myapp  # Both stable and canary have this label

This routes traffic to both versions. But we need to control the percentage.

Step 3: Use a Service Mesh (Istio)

For fine-grained control, use Istio:

yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp
  http:
  - route:
    - destination:
        host: myapp
        subset: stable
      weight: 95
    - destination:
        host: myapp
        subset: canary
      weight: 5
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myapp
spec:
  host: myapp
  subsets:
  - name: stable
    labels:
      version: stable
  - name: canary
    labels:
      version: canary

Using Flagger (Kubernetes Progressive Delivery)

Flagger automates Canary deployments with Istio, Linkerd, or App Mesh.

bash

# Install Flagger
helm repo add flagger https://flagger.app
helm install flagger flagger/flagger

Create a Canary object:

yaml

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  # Deployment reference
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  # Service reference
  service:
    port: 80
    targetPort: 8080
  # Analysis configuration
  analysis:
    interval: 1m
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: request-success-rate
      threshold: 99
      interval: 1m
    - name: request-duration
      threshold: 500
      interval: 1m
    webhooks:
      - name: acceptance-test
        url: http://flagger-loadtester.test/gateway
        timeout: 30s
      - name: load-test
        url: http://flagger-loadtester.test/load
        timeout: 5s

What this does:

• Starts with 0% canary traffic

• Gradually increases to 50% in steps of 5%

• Monitors request success rate and duration

• If metrics drop below thresholds, rollback automatically

• If everything passes, complete the rollout

---

Part 4: Implementing Canary in AWS

AWS (Using Load Balancer Weighted Targets)

Step 1: Create two target groups

hcl

# Blue (stable) target group
resource "aws_lb_target_group" "stable" {
  name = "stable-tg"
  port = 80
  protocol = "HTTP"
  vpc_id = aws_vpc.main.id
}

# Green (canary) target group
resource "aws_lb_target_group" "canary" {
  name = "canary-tg"
  port = 80
  protocol = "HTTP"
  vpc_id = aws_vpc.main.id
}

Step 2: Register instances to target groups

hcl

resource "aws_lb_target_group_attachment" "stable" {
  target_group_arn = aws_lb_target_group.stable.arn
  target_id = aws_instance.stable.id
  port = 80
}

resource "aws_lb_target_group_attachment" "canary" {
  target_group_arn = aws_lb_target_group.canary.arn
  target_id = aws_instance.canary.id
  port = 80
}

Step 3: Configure weighted routing

hcl

# The listener rule can be updated to change weights
resource "aws_lb_listener_rule" "main" {
  listener_arn = aws_lb_listener.main.arn
  priority = 100
  
  action {
    type = "forward"
    forward {
      target_group {
        arn = aws_lb_target_group.stable.arn
        weight = 95
      }
      target_group {
        arn = aws_lb_target_group.canary.arn
        weight = 5
      }
    }
  }
}

Step 4: Update weights gradually

bash

# 5% canary
aws elbv2 modify-rule --rule-arn arn:aws:elasticloadbalancing:... \
  --actions '[{"Type":"forward","ForwardConfig":{"TargetGroups":[{"TargetGroupArn":"stable-tg","Weight":95},{"TargetGroupArn":"canary-tg","Weight":5}]}}]'

# 50% canary
aws elbv2 modify-rule --rule-arn arn:aws:elasticloadbalancing:... \
  --actions '[{"Type":"forward","ForwardConfig":{"TargetGroups":[{"TargetGroupArn":"stable-tg","Weight":50},{"TargetGroupArn":"canary-tg","Weight":50}]}}]'

# 100% canary (decommission stable)
aws elbv2 modify-rule --rule-arn arn:aws:elasticloadbalancing:... \
  --actions '[{"Type":"forward","ForwardConfig":{"TargetGroups":[{"TargetGroupArn":"canary-tg","Weight":100}]}}]'

---

Part 5: Monitoring Canary Health

Key Metrics to Monitor

Application metrics:

• HTTP status codes (200, 404, 500)

• Response time (p50, p95, p99)

• Error rate

• Request count

Infrastructure metrics:

• CPU usage

• Memory usage

• Network I/O

• Disk I/O

Business metrics:

• Conversion rate

• User engagement

• Revenue per user

• Session length

Setting Thresholds

Metric	Warning Threshold	Critical Threshold
Error rate	> 5%	> 10%
Latency (p95)	> 500ms	> 1000ms
CPU usage	> 70%	> 90%
Memory usage	> 80%	> 90%

---

Part 6: Real-World Canary Scenarios

Scenario 1: High-Traffic E-Commerce Site

Challenge: An e-commerce site needs to deploy a new payment integration. Any failure could cause lost revenue.

Solution:

1. Deploy new version to canary (5% of users)

2. Monitor conversion rate in the canary group

3. If conversion rate stays the same, increase to 20%

4. If conversion rate drops, roll back immediately

5. After 24 hours of success, deploy to 100%

Scenario 2: Microservices Update

Challenge: A microservice update changes the API response format. Some downstream services depend on the old format.

Solution:

1. Deploy canary with backward-compatible API (supports both old and new)

2. Route 10% of traffic to canary

3. Monitor downstream services for errors

4. If no errors, increase to 50%

5. If errors appear, roll back and fix API contract

---

Canary Deployment Checklist

Pre-Deployment

• Monitoring is set up to compare canary vs stable

• Rollback procedures are documented

• Canary environment is ready (resources allocated)

• Traffic routing is configured (weighted routing, service mesh)

• Smoke tests are ready to run

• Alerting thresholds are defined

During Deployment

• Initial deployment to canary (0% traffic)

• Run smoke tests on canary

• Route 5% of traffic to canary

• Monitor metrics for 5-10 minutes

• Increase to 20% if metrics are healthy

• Monitor again

• Increase to 50%

• Monitor

• Increase to 100%

• Terminate stable version

Post-Deployment

• Verify all metrics are stable

• Document the rollout

• Clean up old resources

---

Summary

Aspect	Canary
Downtime	Zero
Rollback	Instant (stop routing)
Risk	Low (gradual exposure)
Testing	Real user traffic
A/B testing	Yes
Complexity	High
Tools	Istio, Flagger, Linkerd

Canary deployment is the safest way to deploy new versions to production. You validate the new version with real traffic before exposing it to all users. If something goes wrong, only a small percentage of users are affected, and rollback is immediate.

---

Learn More

Practice Canary deployments with hands-on exercises in our interactive labs:
https://devops.trainwithsky.com/