We Live in a Multi-Cluster Kubernetes World | Rafay
Kubecon 2019 Takeaway Part 1: We Live in a Multi-Cluster World
December 2, 2019
This blog was co-authored by Rupinder (Robbie) Gill and Haseeb Budhani.
Kubecon 2019 was an incredibly engaging event where we had the pleasure of engaging with Kubernetes enthusiasts from a variety of different verticals. Here’s a trend we noticed across the board: Development teams that are early in their Kubernetes journey build out larger clusters and use Kubernetes namespaces to implement multi-tenancy. This seems like a logical choice, given that the namespace concept is designed to do exactly this. But teams that have been at it for some time and have experienced multiple Kubernetes version upgrades tend to spin up many, smaller clusters and choose to group fewer services into the same cluster.
Why the difference in opinion? Experience. Engineers we spoke to shared the following technical reasons (in no particular order) for choosing to go with the many, smaller clusters approach:
- Blast radius: Every time Kubernetes or a supporting component is upgraded, each service may need to be updated to work with the new version. Someone needs to ensure that all services in a given cluster are ready to work with new APIs if older APIs have been deprecated in the new version. This type of upgrade can impact schedules across multiple teams. Best to let teams run smaller clusters where the impact can be broken up into smaller morsels.
- Security requirements: A set of services may have unique hardening and data retention requirements, and it may make sense to deploy these services in a hardened cluster with stringent auditing, auth, and logging policies.
- Scaling requirements: If a few services have massive scaling requirements, it may be best to deploy them into dedicated clusters to protect against other services experiencing “pod pending” events due to busier services taking up an inordinate percentage of available resources.
- Integration requirements: Some services may need a special admission controller, high-speed storage, etc., while others may not. Services that need such integrations may be best grouped together in clusters that are pre-configured with required packages or the right storage class.
- Custom enhancements: Some services may lead the DevOps team to develop enhancements to Kubernetes. To protect against unforeseen side effects from such enhancements, services that need these enhancements can be deployed on customized clusters.
- Network load requirements: Services that are expected to drive high network load are best deployed on separate clusters to protect other services that may get starved otherwise.
Engineers also listed business reasons for running many, smaller clusters:
- Compliance: If end users are distributed globally, it's better to deploy clusters in target geographies to comply with data sovereignty or other regional regulations instead of implementing complex data management strategies centrally.
- Hybrid or multi-cloud strategies: Many companies need to manage a mix of environments for various reasons, such as pre-existing assets, M&A activity, and demanding customers.
- Performance: If the end user population is spread across geographies and the application is designed appropriately, it may make sense to deploy across multiple clusters.
The fact that companies like VMware and Microsoft recently announced tech previews of products to help companies manage clusters across hybrid environments implies they also recognize this trend. At Rafay, we run our SaaS controller as a cloud-native service that leverages a variety of open source and home-grown components. We operate many small clusters. And so should you, if you aren’t doing it already.
In a follow-on blog, we’ll discuss how your peers are not only running multiple clusters but are also leveraging more than one Kubernetes distribution to boot. In the meantime, if you’d like to see how Rafay can help you operate a fleet of clusters across any environment, please feel free to get in touch.