etcd Explained:

 

etcd Explained: Kubernetes' Distributed Brain

📅 Published: September 2026
⏱️ Estimated Reading Time: 14 minutes
🏷️ Tags: etcd, Kubernetes, Distributed Systems, Key-Value Store, DevOps

---

Introduction: What is etcd?

etcd is a distributed, reliable key-value store that serves as the single source of truth for Kubernetes clusters. It stores the entire cluster's configuration, state, and metadata—everything from node information and pod definitions to secrets and ConfigMaps.

Think of etcd as Kubernetes' brain. Just as your brain stores all your memories and controls your body, etcd stores all cluster data and enables the control plane to make decisions. If etcd fails, the cluster loses its memory and cannot function.

Key features:

• Distributed: Runs as a cluster of multiple nodes for high availability

• Consistent: Uses the Raft consensus algorithm to ensure all nodes agree

• Fast: Sub-millisecond read and write latency

• Reliable: Persists data on disk and replicates it across nodes

• Secure: Supports TLS encryption and authentication

---

Part 1: Why etcd Matters in Kubernetes

What etcd Stores in Kubernetes

etcd stores everything in the cluster:

• Nodes: Which nodes exist and their status

• Pods: Which pods are running, where, and their state

• Services: Service definitions and endpoints

• Deployments: Deployment configurations and status

• ConfigMaps: Configuration data

• Secrets: Sensitive information like passwords and API keys

• PersistentVolumes: Storage definitions

• ServiceAccounts: Identity information

• NetworkPolicies: Network security rules

Why etcd is Critical

• Single source of truth: All components read from and write to etcd

• State persistence: Cluster state survives restarts

• Consistency: All nodes see the same data

• Watch mechanism: Components can watch for changes

• API server dependency: The API server can only read from etcd

What Happens if etcd Fails?

• Cluster goes down: No reads or writes possible

• No scheduling: New pods cannot be created

• No updates: Existing resources cannot be modified

• No health checks: API server cannot monitor the cluster

• Cluster recovery: May require restoring from backup

---

Part 2: etcd Architecture

Core Components

text

┌─────────────────────────────────────────────────────────────────┐
│                         etcd Cluster                            │
│                                                                 │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐        │
│  │  etcd Node  │    │  etcd Node  │    │  etcd Node  │        │
│  │   (Leader)  │◀──▶│ (Follower)  │◀──▶│ (Follower)  │        │
│  └─────────────┘    └─────────────┘    └─────────────┘        │
│         │                  │                  │                │
│         └──────────────────┼──────────────────┘                │
│                            │                                    │
│                       ┌─────▼─────┐                           │
│                       │   Raft    │                           │
│                       │ Consensus │                           │
│                       └───────────┘                           │
└─────────────────────────────────────────────────────────────────┘

etcd Cluster Topology

Configuration	Nodes	Fault Tolerance	Use Case
Single Node	1	None	Development, testing
3 Nodes	3	1 failure	Production (minimal)
5 Nodes	5	2 failures	Production (recommended)
7 Nodes	7	3 failures	Large-scale production

Raft Consensus Algorithm

Raft ensures all etcd nodes agree on the cluster state:

1. Leader Election: One node is elected as the leader

2. Log Replication: The leader replicates changes to followers

3. Commitment: A change is committed when a majority of nodes confirm it

4. Safety: Ensures data consistency even with failures

Benefits of Raft:

• Proven algorithm used by many distributed systems

• Simpler than Paxos (easier to understand and implement)

• Provides strong consistency guarantees

• Handles leader failures automatically

---

Part 3: etcd Concepts

Key-Value Store

etcd stores data as hierarchical key-value pairs:

text

/registry/
├── nodes/
│   ├── node1
│   └── node2
├── pods/
│   ├── default/
│   │   ├── my-pod-abc123
│   │   └── my-pod-def456
│   └── kube-system/
│       ├── etcd-0
│       └── apiserver-0
├── services/
│   ├── default/
│   │   └── my-service
│   └── kube-system/
│       └── kube-dns
└── secrets/
    └── default/
        └── my-secret

Watches

etcd supports watch functionality: clients can watch for changes to keys.

bash

# Watch for changes to a key
etcdctl watch /registry/nodes

# Watch with prefix
etcdctl watch /registry/pods/ --prefix

Leases

Leases are time-based contracts that expire automatically.

text

# Create a lease with TTL
etcdctl lease grant 60  # 60-second lease

# Use lease with key
etcdctl put --lease=1234abcd /my-key "value"

Transactions

etcd supports atomic operations with transactions:

bash

# Compare-and-swap
etcdctl txn \
  --compare='value("/my-key")="old"' \
  --success='put "/my-key" "new"' \
  --failure='get "/my-key"'

---

Part 4: etcd Operations

Basic etcdctl Commands

bash

# Set and get keys
etcdctl put /my-key "my-value"
etcdctl get /my-key

# Delete a key
etcdctl del /my-key

# List keys with prefix
etcdctl get /registry/pods/ --prefix

# Watch changes
etcdctl watch /registry/nodes

# Get cluster status
etcdctl endpoint status

# Get cluster health
etcdctl endpoint health

# Check member list
etcdctl member list

Kubernetes-Specific etcd Commands

bash

# List all Kubernetes resources
etcdctl get /registry/ --prefix

# List all nodes
etcdctl get /registry/nodes/ --prefix

# List all pods in default namespace
etcdctl get /registry/pods/default/ --prefix

# Get a specific pod
etcdctl get /registry/pods/default/my-pod-abc123

# Count resources
etcdctl get /registry/ --prefix | wc -l

Backup and Restore

Backup etcd:

bash

# Create a snapshot
ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify snapshot
ETCDCTL_API=3 etcdctl snapshot status snapshot.db

Restore etcd:

bash

# Stop kube-apiserver
systemctl stop kube-apiserver

# Restore from snapshot
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \
  --data-dir=/var/lib/etcd-restore

# Replace data directory
mv /var/lib/etcd /var/lib/etcd-backup
mv /var/lib/etcd-restore /var/lib/etcd

# Restart kube-apiserver
systemctl start kube-apiserver

---

Part 5: etcd Security

Authentication

etcd supports TLS-based authentication:

yaml

# etcd configuration
security:
  authentication:
    - name: "username"
      password: "password"

TLS Encryption

yaml

# etcd configuration
security:
  tls:
    cert-file: /etc/etcd/server.crt
    key-file: /etc/etcd/server.key
    trusted-ca-file: /etc/etcd/ca.crt
    client-cert-auth: true

RBAC (Role-Based Access Control)

etcd supports RBAC for fine-grained access control:

bash

# Create a role
etcdctl role add reader

# Grant permissions
etcdctl role grant-read reader /registry/pods/

# Assign role to user
etcdctl user grant-role user1 reader

---

Part 6: etcd Troubleshooting

Common etcd Issues

Issue	Symptoms	Solution
Node failure	Cluster degraded	Replace failed node
Leader election	Slow writes	Check network latency
Disk full	Writes fail	Free disk space
Certificate expiry	Authentication failures	Renew certificates
Inconsistent state	Different views	Restore from backup

Checking etcd Health

bash

# Check cluster health
etcdctl endpoint health

# Check cluster status
etcdctl endpoint status

# Check disk usage
du -sh /var/lib/etcd

# Check logs
journalctl -u etcd -f

Monitoring etcd

Key metrics to monitor:

Metric	Warning Threshold	Critical Threshold
Leader elections	>1 in 1 hour	>5 in 1 hour
Request latency	>100ms	>500ms
Disk usage	>80%	>90%
Follower lag	>10s	>60s
Database size	>4GB	>8GB

---

Part 7: etcd Best Practices

Deployment Best Practices

1. Run odd number of nodes (3, 5, 7) for quorum

2. Use dedicated storage (SSD) for etcd

3. Deploy across availability zones for high availability

4. Use separate networks for etcd traffic

5. Enable TLS for all communication

6. Regular backups (minimum daily)

7. Monitor resource usage (CPU, memory, disk)

Performance Optimization

• Use SSD storage: Improves I/O performance

• Increase file descriptors: ulimit -n 65536

• Tune etcd parameters: --quota-backend-bytes=8589934592

• Use compression: --auto-compaction-retention=1000

• Monitor database size: Keep it under 8GB

Backup Strategy

text

Scheduled backups:
- Daily snapshot backup
- Weekly disaster recovery (DR) backup
- Monthly archival backup

Retention policy:
- Daily: 7 days
- Weekly: 4 weeks
- Monthly: 6 months

---

etcd Commands Cheat Sheet

bash

# Basic operations
etcdctl put <key> <value>
etcdctl get <key>
etcdctl del <key>
etcdctl watch <key>
etcdctl status

# Cluster management
etcdctl member list
etcdctl member remove <member-id>
etcdctl member add <name> --peer-urls=<url>

# Backup and restore
etcdctl snapshot save <file>
etcdctl snapshot status <file>
etcdctl snapshot restore <file>

# Authentication
etcdctl user add <username>
etcdctl role add <role>
etcdctl user grant-role <username> <role>

---

Summary

Aspect	etcd
Purpose	Distributed key-value store
Role in Kubernetes	Stores cluster state
Consensus	Raft algorithm
Fault tolerance	Majority quorum
Storage	Persistent disk
Security	TLS, RBAC, Authentication
Backup	Snapshot-based

etcd is the foundation of Kubernetes. Understanding and properly managing etcd is essential for any Kubernetes administrator.

---

Learn More

Practice etcd with hands-on exercises in our interactive labs:
https://devops.trainwithsky.com