In the daily life of a Site Reliability Engineer, the main goal is to reduce all the work we call toil. But what is toil? Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly as a service grows. This blog post describes our journey to automate our nodes rotation process when we have a new AMI release and the open source tools we built on this.
Mattermost’s Kubernetes Operator spins up and manages Mattermost instances running on Kubernetes based on a ClusterInstallation Custom Resource (CR). Mattermost Operator 1.0 has evolved a lot since its release, along with the ClusterInstallation CR in the v1alpha version. As time went by — as with any software — the Operator gained more features, configuration options, functionalities, and technical debt.
Since it first appeared in June 2014, Kubernetes has become something of a household name, at least in houses developers live in. The open source container orchestration platform makes challenges like load balancing, secret management, and portability a cinch and makes it easy to orchestrate large, highly scalable and distributed systems.
One of the key benefits of using Kubernetes is that it’s very flexible and fault tolerant. However, that also means that it has quite a lot of complexity to deal with. A well-built operator abstracts that complexity away and helps manage updates and upgrades seamlessly. The Mattermost Kubernetes operator is basically like having a Mattermost Cloud Engineer running inside your Kubernetes cluster managing Mattermost for you.
Kubernetes is said to be the platform to build platforms on. In Freshworks’ case, this is very true. Once Kubernetes established itself as the de-facto container orchestration platform, we set out to create a platform around it with certain very specific capabilities in mind. Kubernetes is a complex beast and takes some effort to tame. Once that is done, there are tremendous advantages.
Basecamp’s newest product HEY has lived on Kubernetes since development first began. While our applications are majestic monoliths, a product like HEY has numerous supporting services that run along-side the main app like our mail pipeline (Postfix and friends), Resque (and Resque Scheduler), and nginx, making Kubernetes a great orchestration option for us.
