Shared clusters

What Will You Do

In this part of the self-paced exercise, we will focus on the following use case:

Clusters are shared between teams. Platform teams want to enforce governance controls but not at the cost of slowing down application teams.

For this example, we will consider two teams, alpha and beta sharing a cluster, shared-cluster.

1. Step 1: Create a "central" project for infrastructure management and to enforce governance (through blueprints etc.)
2. Step 2: Create projects for the two teams (e.g. alpha & beta)
3. Step 3: Import the shared cluster to the "central" project by applying the bootstrap
4. Step 4: Share the imported cluster with projects, alpha and beta. Apply resource quotas
5. Step 5: Add labels to existing namespaces in the imported cluster to align with the teams that own and operate them
6. Step 6: Create a blueprint with "namespace sync on" and apply it to the cluster
7. Step 7: Verify that the namespaces are imported to the right projects and are healthy
8. Step 8: Create "project specific roles" for users in alpha/beta teams and verify access

Declarative specifications are also provided for this exercise.

Step 1: Create a central project for the platform team

• Login into your Org as an Org Admin

• Click on New Project

• Provide the name for the project (e.g. central) and a suitable description

• Click Create

Step 2: Create projects for the two teams

• Click on New Project
• Provide names for the projects (e.g. alpha and beta) and a suitable description
• Click Create

Step 3: Import the shared cluster to the "central" project

• Navigate to project central

• Click on New Cluster

• Select Import Existing Kubernetes Cluster

• Click Continue

• Select Environment and Kubernetes Distribution as appropriate

• Provide a cluster name (e.g. shared-cluster-alpha-beta) and description as needed

• Click Continue

• We will keep the Blueprint selection as minimal for now, click Continue

• Download the cryptographically unique bootstrap YAML required to register the cluster

• Apply bootstrap to the clusters to initiate the import process

Use kubectl to apply the "cluster bootstrap" file on your K8s cluster.

kubectl apply -f  [path to file]/shared-cluster-alpha-beta-bootstrap.yaml

This will create a namespace for the k8s mgmt operator, download the container images, and register with the controller. This one time import process can take ~2 minutes.

namespace/rafay-system created
serviceaccount/system-sa created
clusterrole.rbac.authorization.k8s.io/rafay:manager created
clusterrolebinding.rbac.authorization.k8s.io/rafay:rafay-system:manager-rolebinding created
clusterrole.rbac.authorization.k8s.io/rafay:proxy-role created
clusterrolebinding.rbac.authorization.k8s.io/rafay:rafay-system:proxy-rolebinding created
priorityclass.scheduling.k8s.io/rafay-cluster-critical created
role.rbac.authorization.k8s.io/rafay:leader-election-role created
rolebinding.rbac.authorization.k8s.io/rafay:leader-election-rolebinding created
customresourcedefinition.apiextensions.k8s.io/namespaces.cluster.rafay.dev created
customresourcedefinition.apiextensions.k8s.io/tasklets.cluster.rafay.dev created
customresourcedefinition.apiextensions.k8s.io/tasks.cluster.rafay.dev created
service/controller-manager-metrics-service-v3 created
deployment.apps/controller-manager-v3 created
configmap/connector-config-v3 created
configmap/proxy-config-v3 created
deployment.apps/rafay-connector-v3 created
service/rafay-drift-v3 created
validatingwebhookconfiguration.admissionregistration.k8s.io/rafay-drift-validate-v3 created

Once the k8s operator is operational, it will "establish and maintain a heartbeat" with the controller.

Step 4: Share the imported cluster and apply resource quotas

Share the imported cluster

• Navigate to central project

• Click on the gear icon next to the shared cluster (e.g. shared-cluster-alpha-beta) and select the Manage Cluster Sharing option from the dropdown

• Select the Specific Projects options for Sharing

• Select alpha and beta from the list

• Click Save

Apply resource quotas

• Click on the gear icon next to project alpha in the listing page

• Click on the Resource Quotas tab

• Configure Project Limit (per cluster limit) and Namespace Limit as appropriate

Important: Project Resource quotas ensure that the cluster resource utilization within that specific project context never exceeds the configured limit. This makes sure that teams cannot consume resources (more than what is assigned) and impact other teams with whom the cluster is being shared. Repeat the same exercise for project beta.

Step 5: Add labels to existing namespaces

In this specific example, we have two existing namespaces (alpha1, alpha2) in the shared cluster that belong to the alpha team and two existing namespaces (beta1, beta2) that belong to the beta team. There is also an "infra" namespace where common services such as security tooling etc. have been installed by the platform.

kubectl get ns

NAME              STATUS   AGE
alpha1            Active   89s
alpha2            Active   86s
beta1             Active   82s
beta2             Active   79s
default           Active   62m
infra             Active   66s
kube-node-lease   Active   62m
kube-public       Active   62m
kube-system       Active   62m
rafay-system      Active   25m

The platform offers a namespace reconciliation feature that can synchronize existing namespaces to the platform and convert them into managed namespaces. As part of this process, we would want to make sure that namespaces alpha1/alpha2 get synchronized to alpha project and namespaces beta1/beta2 and beta project.

In order to do this, add k8smgmt.io/project label with the project name to each of the namespaces.

kubectl label ns alpha1 k8smgmt.io/project=alpha
kubectl label ns alpha2 k8smgmt.io/project=alpha
kubectl label ns beta1 k8smgmt.io/project=beta
kubectl label ns beta2 k8smgmt.io/project=beta

If the label is not added, the namespace gets synchronized to the parent project (which is "central" project in this case). We will not add a label to "infra" namespace, it will be synchronized to the central project.

Step 6: Apply a blueprint with "namespace sync on"

• Navigate to the central project

• Click on New Blueprint

• Provide a name (e.g. central-blueprint) and an appropriate description

• Click Save

• Provide a version name (e.g. v1), select minimal as the base blueprint

• Turn the Enable Namespace Sync on

• Click Save Changes

• Navigate to the shared cluster (e.g. shared-cluster-alpha-beta)

• Click on the gear icon next to the cluster and select the Update Blueprint option from the dropdown

• Select the blueprint and the version (e.g. central-blueprint, v1)

• Click Save and Publish

• Wait for the Blueprint sync process to complete

Step 7: Verify that namespaces are imported to the right projects

Navigate to the namespaces page in the central project. You should be able to see the "infra" namespace listed within this project context.

You should be able to similarly see alpha1/alpha2 namespaces in alpha project and beta1/beta2 namespaces in beta project.

Step 8: Create "project specific roles" for users

• Login into your Org as an Org Admin

• Navigate to System -> Users

• Click on New User

• Input appropriate details (do not add any groups as yet)

• Click Save & Exit

• You will see a pop-up stating that the user has not been assigned to any groups. Select Yes to continue

• Select the user that was just created

• Click on the Projects tab

• Select Assign User To Project

• Select alpha project from the dropdown on the left

• Select the Workspace Admin role

• Click Save & Exit

The workspace admin role allows the user to:

• Create a new namespace
• Update resource quotas on existing namespaces
• Delete an existing namespace

All of these actions listed above can be performed within their specific project context (i.e. they cannot delete namespaces belong to other teams in the shared cluster etc.) and is also constrained by the governance controls put in place by the platform teams (e.g. creation of a new namespace is blocked if the project resource quota is exceeded).

Recap

Congratulations! At this point, you have successfully leveraged "Projects" and "Workspace as a service" capabilities to achieve isolation between two teams that share a cluster.