Learn KOP - AKS Add Node Pool - Rafay Product Documentation

Part 3: Node Pool

What Will You Do

In part 3, you will:

Add a new node pool to your AKS cluster increasing its capacity
Remove a node pool from your AKS cluster

Watch a video of this exercise.

Add Node Pool AKS Cluster: Getting Started with Rafay Terraform Provider for AKS - YouTube

Assumptions

This part assumes that you have completed Part 1 of this series and have a successfully provisioned and healthy AKS cluster.

Add Node Pool

In this step, we will scale the cluster by adding a new "spot instance" based node pool to the cluster.

Select a method to provision and manage your AKS cluster from the tabs below.

In this step, we will add a spot instance node pool to the cluster.

Navigate to the previously created project in your Org
Select Infrastructure -> Clusters
Click on the cluster name of the previously created cluster
Click the "Node Pools" tab
Click "Add Node Pool"
Enter a "Name" for the node pool
Select "User" for the "Mode"
Select the "K8s Version" to match the existing node pool
Select "Enable Spot Price"
Enter a "Spot Max Price"
Click "Save & Provision"

From the web console, we can see that the new node pool is being created. This could take up to 15 minutes to complete.

Monitor the web console until the node pool has been created

In this step, we will add a spot instance node pool to the cluster. We will modify the specification file that was applied in step 1.

Add the following node pool configuration code to the previously applied cluster specification file. Note, update the "location" field to match your environment.

- apiVersion: "2022-07-01"
  location: centralindia
  name: pool2
  properties:
    count: 1
    enableAutoScaling: true
    maxCount: 1
    maxPods: 40
    minCount: 1
    mode: User
    orchestratorVersion: 1.23.8
    osType: Linux
    scaleSetPriority: Spot
    spotMaxPrice: 0.03
    type: VirtualMachineScaleSets
    vmSize: Standard_DS2_v2
  type: Microsoft.ContainerService/managedClusters/agentPools

The fully updated cluster specification file including the newly added spot instance node pool code will look like this:

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  name: aks-get-started-cluster
  project: aks
spec:
  blueprint: default-aks
  cloudprovider: Azure-CC
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: aks-get-started-cluster
    spec:
      managedCluster:
        apiVersion: "2022-07-01"
        identity:
          type: SystemAssigned
        location: centralindia
        properties:
          apiServerAccessProfile:
            enablePrivateCluster: true
          dnsPrefix: aks-get-started-cluster-dns
          kubernetesVersion: 1.23.8
          networkProfile:
            loadBalancerSku: standard
            networkPlugin: kubenet
        sku:
          name: Basic
          tier: Free
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2022-07-01"
        location: centralindia
        name: primary
        properties:
          count: 2
          enableAutoScaling: true
          maxCount: 2
          maxPods: 40
          minCount: 2
          mode: System
          orchestratorVersion: 1.23.8
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
        type: Microsoft.ContainerService/managedClusters/agentPools
      - apiVersion: "2022-07-01"
        location: centralindia
        name: pool2
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 40
          minCount: 1
          mode: User
          orchestratorVersion: 1.23.8
          osType: Linux
          scaleSetPriority: Spot
          spotMaxPrice: 0.03
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: Rafay-ResourceGroup
  proxyconfig: {}
  type: aks

Execute the following command to create the spot instance node pool. Note, update the file name in the below command with the name of your updated specification file.

./rctl apply -f aks-cluster-basic.yaml

Expected output (with a task id):

{
  "taskset_id": "3mxzoo2",
  "operations": [
    {
      "operation": "NodegroupCreation",
      "resource_name": "pool2",
      "status": "PROVISION_TASK_STATUS_PENDING"
    },
    {
      "operation": "NodepoolEdit",
      "resource_name": "primary",
      "status": "PROVISION_TASK_STATUS_PENDING"
    }
  ],
  "comments": "The status of the operations can be fetched using taskset_id",
  "status": "PROVISION_TASKSET_STATUS_PENDING"
}

From the web console, we can see that the new node pool is being created. This could take up to 15 minutes to complete.

Monitor the web console until the node pool has been created

To add a node pool:

Edit the terraform.tfvars file. The file location is /terraform/terraform.tfvars.
Copy and paste the following after pool1, then save the file.

  "pool2" = {
    ng_name         = "pool2"
    location        = "<CLUSTER_LOCATION/REGION>"
    node_count      = 1
    node_max_count  = 3
    node_min_count  = 1
    k8s_version     = "<K8S_VERSION>"
    instance_type   = "t3.xlarge"
  }

Open the terminal or command line.
Navigate to the terraform folder.
Run terraform apply. Enter yes when prompted.

It can take 20 minutes to add the node pool to the cluster. Check the console for the node pool status.

Verify Node Pool

Once the new, spot based node pool has been successfully added to the AKS cluster, you should be able to view the "new node pool's details"

Users can also use the Zero Trust Kubectl shell to view details about the new node by using the following command.

kubectl get no

As you can see from this example, Kubectl is reporting "two new nodes" with a relatively recent "age". These are our Spot Instance based nodes from the node pool.

Remove Node Pool

In this step, we will remove the spot instance based node pool we added recently. The process to remove a node pool is very similar to the process of adding a node pool.

In this step, we will remove the spot instance node pool from the cluster.

Navigate to the previously created project in your Org
Select Infrastructure -> Clusters
Click on the cluster name of the previously created cluster
Click the "Node Pools" tab
Click the delete button on the newly created node pool
Click "YES to confirm the node pool deletion

From the web console, we can see that the new node pool is being removed

Monitor the web console until the node pool has been removed. You will only see one node pool remaining.

In this step, we will remove the spot instance node pool from the cluster. We will modify the specification file that was applied in step 2. We will simply remove the code section that was added in step 2 to remove the node pool.

Remove the following node pool configuration code from the previously applied cluster specification file

The updated cluster specification file with the removed spot instance node pool code will look like this:

Execute the following command to remove the spot instance node pool. Note, update the file name in the below command with the name of your updated specification file.

./rctl apply -f aks-cluster-basic.yaml

Expected output (with a task id):

{
  "taskset_id": "dk6z70m",
  "operations": [
    {
      "operation": "NodegroupDeletion",
      "resource_name": "pool2",
      "status": "PROVISION_TASK_STATUS_PENDING"
    }
  ],
  "comments": "The status of the operations can be fetched using taskset_id",
  "status": "PROVISION_TASKSET_STATUS_PENDING"
}

From the web console, we can see that the new node pool is being removed

Monitor the web console until the node pool has been removed. You will only see one node pool remaining.

In this step, we will remove the spot instance node pool from the cluster.

Edit the terraform.tfvars file. The file location is /terraform/terraform.tfvars.
Delete the pool2 configuration, then save the file.
Open the terminal or command line.
Navigate to the terraform folder.
Run terraform apply. Enter yes when prompted.

It can take 20 minutes to remove the node pool to the cluster. Check the console to confirm the node pool has been removed.

Verify Deletion

Once the new, spot based node pool has been successfully deleted from the AKS cluster, you should be able to confirm this on the web console.

Recap

Congratulations!

You have successfully successfully added a "spot instance" based node pool to your AKS cluster to take advantage of discounted compute resources. As a final step, you also successfully removed the "spot" based node pool from your AKS cluster.