etcd is a critical part of Kubernetes—it controls the storage and management of important information that your Kubernetes cluster needs to work properly, such as configuration data, state data, and metadata. It’s a reliable, single source of truth for your Kubernetes cluster data at any point in time.
Since Kubernetes is a distributed system, it makes sense that it uses a distributed data store to ensure reliability, consistency, and the availability of the entire system. This makes etcd the perfect tool for the job.
There are a number of things to be aware of when setting up your etcd cluster, or upgrading your etcd version. In this article, we’ll walk through both of these scenarios.
How does etcd work?
etcd watches for changes to either the actual or desired state of the distributed system. If there are disparities, Kubernetes will make changes to reconcile them. For example, whenever you run kubectl create or kubectl apply, or when there is a crash, you’re making changes to the etcd data store automatically.
etcd is built on top of the Raft algorithm, which is a leader-based distributed system consensus. It consists of a leader node and follower nodes. When the etcd cluster receives a request from the client, the data is updated in all the follower nodes. When most of the follower nodes have been updated, the leader node will send a successful response back to the client.
If the leader node dies, the followers will conduct an election to choose their next leader. If a request is routed to a follower node that has not been updated yet, it will forward the request to the leader node, which will respond with the updated response. This is neatly illustrated in “Understanding Distributed Consensus with Raft” by Kasun Indrasiri.
Setting up an etcd cluster
First, let’s download etcd by running the following command:
It might take a couple of minutes depending on your internet speed. Once it’s done, extract it by running:
Now, you have etcd in your current directory but that’s not where it ought to be, as Kubernetes will be looking for it in /usr/local/bin. So move it into that directory:
Now, you need to create a couple of directories etcd will be looking at for SSL certificates:
If you’re having problems creating the certificates, you can check out this comprehensive guide to help with creating one for yourself.
Next, get your current compute instance IP address to help with dealing with client requests and data transmission with the members of your etcd cluster:
Your etcd name should match the name of the current compute instance, so make it happen by running this line on the terminal:
Create the etcd.service systemd unit file:
Now that your setup is complete, initiate etcd server by running the following commands:
Now, on your different controllers—controller-0, controller-1, controller-2—repeat the previous commands.
You can now go ahead to list the etcd cluster members:
And your output will look like so:
Congratulations, you’ve got your etcd cluster all set up.
Security concerns
Because of the sensitive nature of the data in your etcd cluster, it’s a primary target for hackers. Someone with as much information as is contained in the etcd database can do a lot of havoc with it, such as a ransomware attack, domain takeover, identity theft, and a lot more.
Well, the bad guys will always be there. The most important thing is that you should always be ten steps ahead of them and focus on securing your Kubernetes cluster right now. To secure your etcd cluster, you should:
- Ensure you implement a proper SSL certificate specifically for your etcd instance. etcd enjoys automatic TLS and authentication via certificates from both clients to server and server to server communication. And it’s recommended to create and sign individual certificates for every member of the etcd cluster. This way only your etcd members have your certificate and can communicate to each other securely.
- Encrypt your etcd cluster. Encrypting your etcd cluster means even if your Kubernetes cluster is compromised your private keys will still stay safe. The hacker will need to break into your etcd cluster to gain access to your credentials.
- Authenticate. etcd v2 introduced an authentication API to allow only authorized users to edit key-values in the etc cluster. By default this authentication is turned off. You can turn it on as a root user by running a put request to endpoint
PUT /v2/auth/enable.
Backing up
Backing up your etcd database is as important as backing up your conventional database. In case of unrecoverable system failures, you can always get your system up and running again. Other reasons to back up your etcd could be to replicate your production environment into staging before a major upgrade or a migration from one environment to another.
You can set up a cron job to regularly back up your etcd clusters.
The command below will back up an etcd cluster:
With the previous command, you can back up your etcd cluster using the etcdctl command. You’ll be required to provide your SSL certificate as shown in the snippet above. You already have that, so you’ll have to set the path to the certificate on your instance.
To restore your etcd cluster, it’s recommended that you first stop your etcd cluster, Kube API server and your schedulers, and then run the following commands:
Note the name of the backup file. Make sure the backup is in the directory you are looking to restore it from and the certificate files as well.
Upgrading etcd
etcd has gone through several versions, with each one solving vital bug fixes and adding new features. The current recommended etcd version is 3.x. Depending on the version you’re upgrading from, you might have a little bit of downtime or none at all.
It’s recommended that you migrate between minor versions incrementally. For example, if you are in version 2.0, you should migrate to 2.3 first before migrating to 3.0 or 3.4. This will help reduce the complications that might come with an upgrade.
To upgrade from one version to another, simply modify the command in the etcd manifest file to target the new version and start it:
For pre-upgrade, you’ll need a script to stop etcd, wait until it’s healthy, and then run the official migration script created by the etcd team by running ./etcdctl migrate.
Conclusion
etcd is a core component of Kubernetes. Knowing how to use it, protecting it from the bad guys, upgrading, and backing it up is important, and hopefully you now know how to do all of that. This article should serve as a starting point for your etcd exploration, as there’s a lot more to learn. You can read more about etcd by looking at the etcd documentation.
If you're looking to build a dashboard for monitoring your Kubernetes clusters, then sign up for Airplane. With Airplane, you can transform scripts, queries, APIs, and more into powerful workflows and UIs. Airplane also offers strong built-ins, such as audit logs, permissions settings, approval flows, and more.
To try it out, sign up for a free account or book a demo.
