The Nightmare of Scaling Microservices
Imagine managing 20 microservices. Suddenly, a Database IP changes or you need to update a critical API Key immediately. The old-fashioned way involves logging into each service, editing the .env file, and restarting all 20 containers.
This approach is both time-consuming and risky. Forgetting to update just one service can break the entire system. I once had to stay up until 2 AM checking every single config file because I missed one service in the cluster. It was not a pleasant experience at all.
Why MySQL or Redis Aren’t the Best Choices
When facing this problem, many developers immediately think of using MySQL or Redis to store configurations. However, several issues arise in practice:
- MySQL/PostgreSQL: It’s like using a truck to deliver a single letter. Maintaining hundreds of connections just to fetch a few lines of config is a massive waste of resources.
- Redis: While very fast, it prioritizes performance over consistency by default. In configuration management, consistency is king. You definitely don’t want Service A to receive an old config while Service B has already updated to the new one.
- Physical files: Syncing files across dozens of different servers is an operational nightmare.
etcd – The “Brain” of Distributed Systems
etcd was born to solve these exact problems. It is a distributed key-value store, written in Go, using the Raft consensus algorithm. It ensures data is always consistent across every node. Did you know? etcd is the heart of Kubernetes, where the entire cluster state is stored.
The biggest selling point of etcd is its Watch capability. Instead of a service constantly asking “Is there anything new?”, etcd proactively pushes notifications as soon as a change occurs. This allows applications to update configurations almost instantly (real-time).
How to Install etcd on Linux
We will install the binary version of etcd on Linux (Ubuntu/CentOS). This is the fastest way to get familiar with it before deploying more complex models on K8s.
# Download version v3.5.0
ETCD_VER=v3.5.0
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
# Move to system directory
sudo mv /tmp/etcd-download-test/etcd /usr/local/bin/
sudo mv /tmp/etcd-download-test/etcdctl /usr/local/bin/Check the installation with the command etcd --version. To start experimenting, you can simply run a single node with the command:
etcdAt this point, etcd will listen on port 2379 (for clients) and 2380 (for node-to-node communication).
Data Operations with etcdctl
We use the etcdctl tool to interact with the DB. One vital note: etcd has two API versions, v2 and v3. Always set the following environment variable to use v3 (the current standard):
export ETCDCTL_API=31. Adding and Retrieving Data
Use put to save and get to retrieve:
# Save config
etcdctl put /configs/db_url "postgres://user:pass@localhost:5432/mydb"
# Get value
etcdctl get /configs/db_url2. Watching for Changes (Watch)
Open two terminal windows to see the magic. In terminal 1, run the watch command:
etcdctl watch /configs/api_keyIn terminal 2, update the value:
etcdctl put /configs/api_key "new-secret-key"Terminal 1 will show the new value immediately. In your code (Go or Node.js), you can use a callback function to automatically update environment variables without restarting the app.
Application for Service Discovery
Think of Service Discovery as an automated phonebook. When Service B initializes, it registers itself with etcd with a specific Time To Live (TTL) using the Lease mechanism.
