KOP AKS Clusters - Gitops Examples - Rafay Product Documentation

Examples

Infra GitOps using RCTL

Here are some of the examples to scale an existing cluster through Infra GitOps using RCTL

Below is a cluster config sample yaml file using System-assigned managed identity which is by default and in this azure automatically creates an identity for the AKS cluster and assigns it to the underlying Azure resources.

System-assigned managed identity Example

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  name: demo-akscluster
  project: defaultproject
spec:
  blueprint: default-aks
  blueprintversion: latest
  cloudprovider: akscredentials
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: demo-akscluster
    spec:
      managedCluster:
        additionalMetadata:
          acrProfile:
            acrName: demo_registry
            resourceGroupName: demoresoursegroup
        apiVersion: "2021-05-01"
        identity:
          type: SystemAssigned
        location: eastus2
        properties:
          apiServerAccessProfile:
            enablePrivateCluster: false
          dnsPrefix: demo-akscluster-dns
          kubernetesVersion: 1.21.2
          networkProfile:
            loadBalancerSku: standard
            networkPlugin: kubenet
            networkPolicy: calico
        sku:
          name: Basic
          tier: Free
        tags:
          role: demo
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2021-05-01"
        name: primary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 2
          maxPods: 40
          minCount: 1
          mode: System
          nodeLabels:
            testdemo: demoworker
          orchestratorVersion: 1.21.2
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demoresoursegroup
  proxyconfig: {}
  type: aks

Command to create a cluster using the config file:

./rctl apply -f <config-file.yaml>

Example:

./rctl apply -f demo-akscluster.yaml

Expected Output (with a task id):

{
  "taskset_id": "dk3lekn",
  "operations": [
    {
      "operation": "NodegroupCreation",
      "resource_name": "primary",
      "status": "PROVISION_TASK_STATUS_PENDING"
    },
    {
      "operation": "ClusterCreation",
      "resource_name": "demo-akscluster",
      "status": "PROVISION_TASK_STATUS_PENDING"
    }
  ],
  "comments": "The status of the operations can be fetched using taskset_id",
  "status": "PROVISION_TASKSET_STATUS_PENDING"
}

To know the status of the cluster creation operation, enter the below command with the generated task id

./rctl status apply dk3lekn

Expected Output

{
  "taskset_id": "dk3lekn",
  "operations": [
  {
    "operation": "NodegroupCreation",
    "resource_name": "resource_1",
    "status": "PROVISION_TASK_STATUS_PENDING"
  },
  {
    "operation": "ClusterCreation",
    "resource_name": "demo-akscluster",
    "status": "PROVISION_TASK_STATUS_PENDING"
  }
  ],
  "comments": "Configuration is applied to the cluster successfully",
  "status": "PROVISION_TASKSET_STATUS_COMPLETE"
}

Use the following command to view the status of a cluster-level operation (TaskSet):

./rctl get clustertaskset <cluster_name>

This command can be used to retrieve the status of the last taskset applied on the cluster. This is useful for tracking long-running operations or debugging provisioning issues.

Example

./rctl get clustertaskset demo-akscluster
{
  "tasksetId": "dk3lekn",
  "tasksetOperations": [
    {
      "operationName": "NodeAddition",
      "resourceName": "demo-akscluster",
      "operationStatus": "PROVISION_TASK_STATUS_SUCCESS"
    }
  ],
  "tasksetStatus": "PROVISION_TASKSET_STATUS_COMPLETE",
  "comments": "Node add operation was performed on: demo-802. Configuration is applied to the cluster successfully"
}

Description

tasksetId: Unique identifier for the provisioning taskset
tasksetOperations: Lists each individual operation under the taskset. In this case, it shows a NodeAddition operation performed on the cluster demo-akscluster
operationStatus: Status of the operation. PROVISION_TASK_STATUS_SUCCESS indicates it completed successfully
tasksetStatus: Indicates the overall status of the taskset. PROVISION_TASKSET_STATUS_COMPLETE confirms that all tasks have completed
comments: Provides additional context or results from the operation. In this example, configuration was successfully applied to the cluster

User-Assigned Managed Identity Example

Below is an example YAML configuration file for a reference AKS cluster that utilizes User-Assigned Managed Identity. With user-assigned managed identities, you have the flexibility to create and manage identities independently from the AKS cluster. These identities can then be associated with one or more AKS clusters, enabling seamless identity reuse across multiple AKS clusters or other Azure resources.

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  name: azure-aks-demo
  project: demo
spec:
  blueprint: minimal
  cloudprovider: azure
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: azure-aks-demo
    spec:
      managedCluster:
        apiVersion: "2022-07-01"
        identity:
          type: UserAssigned
          userAssignedIdentities:
            ? /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo/providers/Microsoft.ManagedIdentity/userAssignedIdentities/demo-mgi-cli
            : {}
        location: centralindia
        properties:
          apiServerAccessProfile:
            enablePrivateCluster: true
          dnsPrefix: azure-aks-demo-dns
          kubernetesVersion: 1.26.0
          networkProfile:
            dnsServiceIP: 10.1.0.10
            dockerBridgeCidr: 172.17.0.1/16
            loadBalancerSku: standard
            networkPlugin: azure
            serviceCidr: 10.1.0.0/16
        sku:
          name: Basic
          tier: Free
        tags:
          email: demo@rafay.co
          env: qa
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2022-07-01"
        name: primary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 110
          minCount: 1
          mode: System
          orchestratorVersion: 1.26.0
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
          vnetSubnetID: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo/providers/Microsoft.Network/virtualNetworks/demo-vnet1/subnets/default
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demo
  proxyconfig: {}
  type: aks

In this example, we have configured the AKS cluster to use a User Assigned Managed Identity.

type: UserAssigned: This line indicates that a User Assigned Managed Identity is being used for the AKS cluster.
userAssignedIdentities: This specifies the path to the User Assigned Managed Identity that will be associated with the AKS cluster. In this case, the identity is located at /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo/providers/Microsoft.ManagedIdentity/userAssignedIdentities/demo-mgi-cli: {}

To create a cluster using this config file, you can use the same rctl apply command.

Azure CNI Overlay Example

apiVersion: infra.k8smgmt.io/v3
kind: Cluster
metadata:
  modifiedAt: "2024-05-14T05:34:52.871381Z"
  name: demo-overlay-aks
  project: default
spec:
  blueprintConfig:
    name: minimal
  cloudCredentials: azure
  config:
    kind: aksClusterConfig
    metadata:
      name: demo-overlay-aks
    spec:
      managedCluster:
        apiVersion: "2024-01-01"
        identity:
          type: SystemAssigned
        location: centralindia
        properties:
          apiServerAccessProfile:
            enablePrivateCluster: true
          dnsPrefix: demo-overlay-aks-dns
          enableRBAC: true
          kubernetesVersion: 1.29.4
          networkProfile:
            dnsServiceIP: 10.0.0.10
            loadBalancerSku: standard
            networkPlugin: azure
            networkPluginMode: overlay
            podCidr: 10.244.0.0/16
            serviceCidr: 10.0.0.0/16
          powerState:
            code: Running
        sku:
          name: Base
          tier: Free
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2024-01-01"
        name: primary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 110
          minCount: 1
          mode: System
          orchestratorVersion: 1.29.4
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_B4ms
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demo-rg
  type: aks

In this example, we have configured the Azure CNI Overlay.

apiVersion: "2024-01-01": This parameter should be set to a date on or after January 1, 2024, when configuring Azure CNI in overlay mode
networkPluginMode: overlay: This line indicates that the network plugin mode for the Azure CNI is set to "overlay"
podCidr: 192.168.0.0/16:: podCidr is mandatory when configuring Azure CNI to overlay mode. This parameter specifies the CIDR block that will be used for assigning IP addresses to pods within the overlay network

Important

After the release of apiVersion 2023-02-01, SKU Tier Paid is deprecated with Premium. For older clusters, the previously configured Tier will remain visible in the configuration specifications.
After the release of apiVersion 2023-02-01, the dockerBridgeCIDR setting in network configurations of managedClusters is deprecated. For older clusters, the previously configured dockerBridgeCIDR will remain visible in the configuration specifications.
Ensure that the apiVersion in the managedCluster configuration matches the apiVersion in the nodepool configuration to prevent potential failures during nodepool operations.

Node Auto Provisioning

Node Auto Provisioning (NAP) in Azure Kubernetes Service (AKS) automatically provisions and manages nodes based on workload resource requirements. When enabled, the platform dynamically creates and manages node pools to ensure workloads are scheduled efficiently without manual node pool sizing or scaling. NAP uses the nodeProvisioningProfile configuration in the cluster specification to control automatic node pool management.

Note

Node Auto Provisioning is supported only through RCTL, Terraform v3, and GitOps workflows.

The following example shows how to configure Node Auto Provisioning in the AKS cluster specification.

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  name: demo-auto-nodeprov
  project: default
spec:
  blueprint: default-aks
  cloudprovider: demo_azure2
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: demo-auto-nodeprov
    spec:
      managedCluster:
        additionalMetadata:
          acrProfile:
            registries:
            - acrName: demofnfacr
              resourceGroupName: demo-fnf-cluster-rg
        apiVersion: "2025-10-01"
        identity:
          type: UserAssigned
          userAssignedIdentities:
            ? /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-fnf-cluster-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/demo-fnf-cluster-identity
            : {}
        location: centralindia
        properties:
          aadProfile:
            enableAzureRBAC: true
            managed: true
          addonProfiles:
            azureKeyvaultSecretsProvider:
              config:
                enableSecretRotation: "true"
                rotationPollInterval: 2m
              enabled: true
            azurePolicy:
              enabled: true
            omsAgent:
              config:
                logAnalyticsWorkspaceResourceID: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourcegroups/defaultresourcegroup-cin/providers/microsoft.operationalinsights/workspaces/DefaultWorkspace-a2252eb2-7a25-432b-a5ec-e18eba6f26b1-CIN
              enabled: false
          apiServerAccessProfile:
            enablePrivateCluster: false
            enablePrivateClusterPublicFQDN: false
          autoUpgradeProfile:
            nodeOsUpgradeChannel: None
            upgradeChannel: none
          disableLocalAccounts: true
          dnsPrefix: aks-333-dns
          enableRBAC: true
          identityProfile:
            kubeletIdentity:
              resourceId: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-fnf-cluster-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/demo-kubelet-identity
          ingressProfile:
            webAppRouting:
              dnsZoneResourceIds:
              - /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-rg/providers/Microsoft.Network/dnsZones/demozone.com
              enabled: false
          kubernetesVersion: 1.32.3
          linuxProfile:
            adminUsername: adminuser
            ssh:
              publicKeys:
              - keyData: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsNWPqLqjrIPLjO9CK4xhcrIPyk82wjB+BSWWUwVniSKHJjq3TmZnA1YKSaraGfJZfyQppK6AyqhxYw5l2SLHD896VyZw88p11qC2VBCKoQxn2fLOLJkRumCAclhC9mfMITyF/O5eLXzZEymts3SV160UAKvzPlqigzwdfxwPRDjawCRdKkoB/d1gbPu4qCaNm7B6pf6iG0TLIZXVWkX/GkPXZh40aBq/fU8ou2W87Ew3bK3AznXnN+t2k7t25sriz3sQUzpe7yaWUjnh5KUZ+fmuDsWpXj0cugorltYLFDKpoACKR99cQDoGcVS3RzYIFWLKto/0Ok4TVbL/5a2wrqfaxdJiLyTkzNeKDdd5/iQpSK+DN82iiONET4pHKI6zQ7/LcNzsAC7QrL7w+7F4VYqIqzKOfLpFNA4RjcUMCu/N6JK/kIHjW3cM2PTfZOBleZ7zfOfigTx2E7d+dTbC+ARjSbamJRwrnPFtumr4qVSsAm+H6Xn3YaqN4Tz05h/E=
                  generated-by-azure
          networkProfile:
            dnsServiceIP: 10.0.0.10
            loadBalancerSku: standard
            networkDataplane: cilium
            networkPlugin: azure
            networkPolicy: cilium
            podCidr: 192.168.0.0/16
            serviceCidr: 10.0.0.0/16
          nodeProvisioningProfile:
            defaultNodePools: Auto
            mode: Auto
          nodeResourceGroup: demo-auto-nodeprov
          oidcIssuerProfile:
            enabled: true
          powerState:
            code: Running
          securityProfile:
            workloadIdentity:
              enabled: true
          serviceMeshProfile:
            istio:
              components:
                ingressGateways:
                - enabled: true
                  mode: Internal
              revisions:
              - asm-1-27
            mode: Istio
        sku:
          name: Base
          tier: Free
        tags:
          email: user1@rafay.co
          env: dev
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2023-07-01"
        name: secs222
        properties:
          count: 1
          creationData:
            sourceResourceId: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-rg/providers/Microsoft.ContainerService/snapshots/asdasd26jan
          enableAutoScaling: false
          kubeletConfig:
            containerLogMaxFiles: 2
          maxPods: 110
          mode: System
          nodeTaints:
          - app22=infr22a:PreferNoSchedule
          orchestratorVersion: 1.32.3
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_B4ms
          vnetSubnetID: /subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-fnf-nw-rg/providers/Microsoft.Network/virtualNetworks/demo-fnf-vnet/subnets/default2
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demo-rg
  type: aks

In this example:

mode: Auto enables automatic node provisioning.
defaultNodePools: Auto allows the platform to manage default node pools automatically based on workload requirements.

Limitations

User cannot enable Node Auto Provisioning (NAP) on clusters with autoscaler enabled.
NAP does not support Windows node pools.
IPv6 clusters are not supported.
DiskEncryptionSetId is not allowed for NAP-enabled clusters.
Cluster Stop operation is not supported for NAP-enabled clusters.
Day-2 operations for egress outbound type are not supported for NAP-enabled clusters.
loadBalancerSku must be set to Standard when NAP is enabled with a custom VNet.

Note on Autoscaling and Node Auto Provisioning

When enabling Node Auto Provisioning (NAP), node pools should be aligned with NAP requirements during configuration planning.

If autoscaling is already enabled on an existing node pool and NAP needs to be enabled:

Decommission the existing node pool.
Create a new node pool configured for NAP requirements.
Configure NAP using the newly created node pool.

This approach ensures compatibility with Day-2 operations and maintains consistent node provisioning behavior.

For more details, refer to the AKS Node Auto Provisioning Documentation

Cluster Sharing

For cluster sharing, add a new block to the cluster config (Rafay Spec) as highlighted in the below config file

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  name: aks-democluster
  project: defaultproject
spec:
  blueprint: bp-aks
  blueprintversion: v5
  cloudprovider: cp_aks
  sharing:
      enabled: true
      projects:
      - name: "demoproject1"
      - name: "demoproject2"
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: aks-democluster
    spec:
      managedCluster:
        apiVersion: "2021-05-01"
        identity:
          type: SystemAssigned
        location: centralindia
        properties:
          apiServerAccessProfile:
            enablePrivateCluster: false
          dnsPrefix: aks-dns-demo
          kubernetesVersion: 1.22.11
          networkProfile:
            loadBalancerSku: standard
            networkPlugin: kubenet
        sku:
          name: Basic
          tier: Free
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2021-05-01"
        name: primary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 40
          minCount: 1
          mode: System
          orchestratorVersion: 1.22.11
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demo_group
  proxyconfig: {}
  type: aks

You can also use the wildcard operator "*" to share the cluster across projects

sharing:
    enabled: true
    projects:
    - name: "*"

Notes: When passing the wildcard operator, users cannot pass other projects name

To remove any cluster sharing from the project(s), remove that specific project name(s) and run the apply command

Update Blueprint

Make the required change for Blueprint and use the command

./rctl apply -f demo-akscluster.yaml

Expected output (with a task id):

{
  "taskset_id": "g29wek0",
  "operations": [
    {
      "operation": "BlueprintUpdation",
      "resource_name": "demo-akscluster",
      "status": "PROVISION_TASK_STATUS_PENDING"
    }
  ],
  "comments": "The status of the operations can be fetched using taskset_id",
  "status": "PROVISION_TASKSET_STATUS_PENDING"
}

To know the status of the Blueprint apply operation, enter the below command with the generated task id

./rctl status apply g29wek0

Expected Output

{
  "taskset_id": "g29wek0",
  "operations": [
    {
      "operation": "BlueprintUpdation",
      "resource_name": "demo-akscluster",
      "status": "PROVISION_TASK_STATUS_SUCCESS"
    }
  ],
  "comments": "Configuration is applied to the cluster successfully",
  "status": "PROVISION_TASKSET_STATUS_COMPLETE"
}

Update Cloud Credential

Make the required change for Cloud Credential and use the command

./rctl apply -f demo-akscluster.yaml

Expected output (with a task id):

{
  "taskset_id": "j2q9jm9",
  "operations": [
    {
      "operation": "CloudProviderUpdation",
      "resource_name": "demo-akscluster",
      "status": "PROVISION_TASK_STATUS_PENDING"
    }
  ],
  "comments": "The status of the operations can be fetched using taskset_id",
  "status": "PROVISION_TASKSET_STATUS_PENDING"
}

To know the status of the Cloud Credential apply operation, enter the below command with the generated task id

./rctl status apply j2q9jm9

Expected output

{
  "taskset_id": "j2q9jm9",
  "operations": [
    {
      "operation": "CloudProviderUpdation",
      "resource_name": "demo-akscluster",
      "status": "PROVISION_TASK_STATUS_SUCCESS"
    }
  ],
  "comments": "Configuration is applied to the cluster successfully",
  "status": "PROVISION_TASKSET_STATUS_COMPLETE"
}

Cluster Labels

Users can update Cluster Labels via RCTL using the below Cluster Configuration Yaml file

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  labels:
    newrole: cluslab1
    roles: worker1
  name: newrole_1
  project: defaultproject
spec:
  blueprint: default-aks
  blueprintversion: latest
  cloudprovider: aks-cloudcred
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: demo-akscluster
...

Command to apply the labels to the cluster:

./rctl apply -f <cluster-config.yaml>

Example:

./rctl apply -f demo-akscluster.yaml

Expected output (with a task id):

{
 "taskset_id": "lk5dwme",
 "operations": [
  {
   "operation": "ClusterLabelsUpdation",
   "resource_name": "demo-akscluster",
   "status": "PROVISION_TASK_STATUS_PENDING"
  }
 ],
 "comments": "The status of the operations can be fetched using taskset_id",
 "status": "PROVISION_TASKSET_STATUS_PENDING"
}

To know the status of the Cluster Label apply operation, enter the below command with the generated task id

./rctl status apply lk5dwme

Expected output:

{
  "taskset_id": "lk5dwme",
  "operations": [
    {
      "operation": "ClusterLabelsUpdation",
      "resource_name": "demo-akscluster",
      "status": "PROVISION_TASK_STATUS_SUCCESS"
    }
  ],
  "comments": "Configuration is applied to the cluster successfully",
  "status": "PROVISION_TASKSET_STATUS_COMPLETE"
}