KOP AKS Clusters - Workload Identity - Rafay Product Documentation

Workload Identity Overview & Configuration

Overview

Workload identity allows Kubernetes workloads to securely access Azure Active Directory (Azure AD) protected resources such as Azure Key Vault, Azure Storage, Azure SQL Database, Azure Monitor, and Azure Cosmos DB. This identity solution leverages Azure AD-based managed identities via a workload identity workflow. Azure AD Workload Identity overcomes the limitations of the deprecated Azure AD Pod Identity, such as scale and performance issues with identity assignment.

Users can create a workload identity during cluster provisioning (Day 0). For existing clusters, the workload identity can be created on Day 2 to integrate with the existing infrastructure.

Did you know ?

Workload Identity can be managed through various methods:

Pre-requisite

A Kubernetes cluster with version ≥ v1.20
AKS Workload Identity is supported for existing clusters (Day 2 operations) with an API version of 2023-03-01 or earlier that are already present on the platform. However, for new clusters (Day 0 operations) requiring Workload Identity support, an API version of 2023-03-01 or later is mandatory, as clusters with Workload Identity or OIDC feature flags enabled and an API version earlier than 2023-03-01 are not supported
For existing clusters with an older API version, users can still leverage it by using the workload identity configuration provided below

User vs. Rafay-Managed Identity and Role Assignment

Flow 1: User-Provided Identity and Role Assignments

In this flow, users interact with Rafay by providing managed identity and role-related details. There are several scenarios to consider:

Scenario 1
User Provides Identity Name and Status Attributes

createIdentity=False is set: Rafay does not create the identity. Instead, it works with the provided identity

Scenario 2
User Shares Identity Specification Parameters

createIdentity=True is set: Rafay creates the identity and works with the newly created identity

Scenario 3
User provides Role Assignment specification list

Rafay performs the necessary role assignments on behalf of the user, provided that the relevant permissions (Role Assignment Write and Delete Permissions) are assigned to the Service Principal Name (SPN) used to create the cloud credentials for the request. This is crucial for enabling role assignment under Workload Identity support
Details Provided: Role assignment specification list, including:
- Role ID and Role Name: Either the specific identifier (Role ID) or the name of the role
- Scope: The resource that the managed identity needs to access from within the cluster using the service account. This defines the permissions and boundaries applicable for the resource

Scenario 4
User does not provide Role Assignment list

Rafay assumes that the identity already has the necessary role assignments and does not perform any role assignments

Scenario 5
User provides Service Account as specification parameters

createAccount=True is set: Rafay creates the service account.

Scenario 6
User provides Service Account as reference attributes

createAccount=False is set: Rafay does not create the service account but references the existing service account.

Important

A single workload identity can be associated with multiple service accounts, but one service account cannot be associated with more than one workload identity.

Flow 2: Rafay-Created Identity, Role Assignments and Service Accounts

In this flow, Rafay is responsible for creating the identity, assigning roles, and managing service accounts. There are a few scenarios here as well:

Scenario 1
Rafay creates Workload Identity and assigns roles

Identity Creation: Rafay establishes a new identity
Role Assignments: Rafay applies roles as part of the identity creation process

Scenario 2
Service Account details provided by users

Users supply detailed information for service accounts, and Rafay uses these details to create the service accounts as needed.
createAccount=True is set: Rafay creates the service account.

Important

Workload Identity supports the creation of service accounts. However, automount service accounts are not supported.

Workload Identity Via RCTL

V3 Config Spec (Recommended)

Below is an AKS configuration specification with workload identity parameters, role assignments, and a service account.

apiVersion: infra.k8smgmt.io/v3
kind: Cluster
metadata:
  name: demo-aks-cluster
  project: defaultproject
spec:
  blueprintConfig:
    name: minimal
  cloudCredentials: demo-azure-creds
  type: aks
  config:
    kind: aksClusterConfig
    metadata:
      name: demo-aks-cluster
    spec:
      managedCluster:
        apiVersion: '2024-01-01'
        identity:
          type: SystemAssigned
        location: centralindia
        properties:
          apiServerAccessProfile:
            enablePrivateCluster: true
            privateDNSZone: System
          dnsPrefix: demo-test-dns
          enableRBAC: true
          kubernetesVersion: 1.28.9
          networkProfile:
            loadBalancerSku: Standard
            networkPlugin: kubenet
          powerState:
            code: Stopped
          securityProfile:
            workloadIdentity:
              enabled: true
          oidcIssuerProfile:
            enabled: true
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - type: Microsoft.ContainerService/managedClusters/agentPools
        apiVersion: '2023-11-01'
        name: primary
        location: centralindia
        properties:
          count: 2
          enableAutoScaling: true
          maxCount: 2
          maxPods: 40
          minCount: 1
          mode: System
          orchestratorVersion: 1.28.9
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
      resourceGroupName: demo-rg-ci
      workloadIdentities:
      - createIdentity: false
        metadata:
          clientId: 27514982-9cca-43f6-996d-100b5a641beb
          principalId: 72308b4b-5f25-4359-b52d-07c9b0891141
          name: demo-uai-1
          location: centralindia
          resourceGroup: demo-rg-ci
          tags:
            example.com/owner: "demo"
            example.com/department: "demo"
        roleAssignments:
        - name: "Key Vault Secrets User"
          roleDefinitionId: "8e3af657-a8ff-443c-a75c-2fe8c4bcb635"
          scope: "/subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-rg-ci/providers/Microsoft.KeyVault/vaults/demo-keyvault"
        serviceAccounts:
        - createAccount: false
          metadata:
            name: demo-aks-cluster-sa
            namespace: default
            annotations:
              example.com/role: "demo-aks-cluster-sa"
            labels:
              example.com/owner: "demo"
              example.com/department: "demo"

V1 Config Spec

apiVersion: rafay.io/v1alpha1
kind: Cluster
metadata:
  name: demo-aks-cluster
  project: demo
spec:
  blueprint: minimal
  cloudprovider: demo-azure-creds
  clusterConfig:
    apiVersion: rafay.io/v1alpha1
    kind: aksClusterConfig
    metadata:
      name: demo-aks-cluster
    spec:
      managedCluster:
        apiVersion: "2022-07-01"
        identity:
          type: SystemAssigned
        location: centralindia
        properties:
          apiServerAccessProfile:
            enablePrivateCluster: true
          dnsPrefix: demo-aks-cluster-dns
          enableRBAC: true
          kubernetesVersion: 1.27.7
          networkProfile:
            loadBalancerSku: standard
            networkPlugin: kubenet
          powerState:
            code: Running
          oidcIssuerProfile:
            enabled: true
          securityProfile:
            workloadIdentity:
              enabled: true
        sku:
          name: Basic
          tier: Free
        type: Microsoft.ContainerService/managedClusters
      nodePools:
      - apiVersion: "2022-07-01"
        location: centralindia
        name: primary
        properties:
          count: 1
          enableAutoScaling: true
          maxCount: 1
          maxPods: 110
          minCount: 1
          mode: System
          orchestratorVersion: 1.27.7
          osType: Linux
          type: VirtualMachineScaleSets
          vmSize: Standard_DS2_v2
        type: Microsoft.ContainerService/managedClusters/agentPools
      resourceGroupName: demo-rg-ci
      workloadIdentities:
      - createIdentity: false
        metadata:
          clientId: 27514982-9cca-43f6-996d-100b5a641beb
          principalId: 72308b4b-5f25-4359-b52d-07c9b0891141
          name: demo-uai-1
          location: centralindia
          resourceGroup: demo-rg-ci
          tags:
            example.com/owner: "demo"
            example.com/department: "demo"
        roleAssignments:
        - name: "Key Vault Secrets User"
          roleDefinitionId: "8e3af657-a8ff-443c-a75c-2fe8c4bcb635"
          scope: "/subscriptions/a2252eb2-7a25-432b-a5ec-e18eba6f26b1/resourceGroups/demo-rg-ci/providers/Microsoft.KeyVault/vaults/demo-keyvault"
        serviceAccounts:
        - createAccount: false
          metadata:
            name: demo-aks-cluster-sa
            namespace: default
            annotations:
              example.com/role: "demo-aks-cluster-sa"
            labels:
              example.com/owner: "demo"
              example.com/department: "demo"
  type: aks

Workload Identity Via UI

Workload Identity can be configured through the UI during the cluster creation process by providing the required details in the Workload Identity Settings section.

Enable Enable Workload Identity and Enable OIDC Issuer Profile. Once these options are enabled, the + Workload Identity button becomes active
Click + Workload Identity to configure the workload identity

New Workload Identity

Select Create New
Specify a name, resource group, and select a location
Optionally, add tags as needed
Click + Add Role Assignment and specify the name and scope
Click Save & Continue to proceed
Click + Add Service Account and choose Create New or Use Existing, based on the requirement
Enter the name and namespace
Optionally, add labels and annotations
Click Save

Use Existing Workload Identity

Select Use Existing
Specify the name, resource group, and location
Enter the Client ID and Principal ID
Optionally, add tags if required
Similar to the Create New procedure:
- Click Add Role Assignment
- Click + Add Service Account:
  - Choose Create New or Use Existing, based on the requirement
  - Enter the name and namespace
  - Optionally, add labels and annotations
Click Save

Save & Customize Option

Workload Identity configuration parameters can also be added using the Save & Customize option if they are not included in the Workload Identity form available in the UI. During the cluster creation process, enter the required details and click the Save & Customize button. After providing the workload identity configuration specification, as shown in the example below, click Save Changes.

⚠️ Important:
Once the workload identity is deployed, changes to the workload identity configuration will not automatically reflect in existing workloads. To apply the new identity settings and ensure that all configurations are up-to-date, affected workloads will need to be redeployed. This process involves updating and redeploying K8s pods or deployments to apply the new workload identity settings.

Workload Identity Takeover as Part of AKS Cluster Takeover

Workload identity takeover ensures seamless access and continuity by transferring workload identities between AKS clusters during migration or conversion, preserving configurations and permissions.

Currently, workload identity takeover is supported only if the associated service accounts have been discovered
Workload identity Day 2 operations will function on a cluster that has undergone a takeover

Once workload identity is deployed, changes to the workload identity configuration will not automatically reflect in existing workloads. To apply the new identity settings and ensure that all configurations are up-to-date, affected workloads will need to be redeployed. This process involves updating and redeploying Kubernetes pods or deployments to apply the new workload identity settings.

Note: Support for service account takeover and workload identity takeover is planned for an upcoming release.

Alternative Path for Users

There is an alternative path currently available that ensures customers can take part of transition:

Take Over the Cluster with Workload Identity Feature Flags: Users can take over the cluster with the workload identity (WI) feature flags either enabled or disabled.
Enable Workload Identity Feature Post Takeover: After taking over the cluster, users can enable the WI feature if it was previously disabled.
Soft Create Workload Identities: Users can soft create the workload identities using metadata from the existing workload identities (WIs).
Day 2 Operations Availability: All Day 2 operations that are available with Rafay created clusters are also available with the taken over cluster.

This alternative path allows for flexibility and continuity in managing workload identities during cluster transition.

To learn more about the use cases and best practices, you can refer to the Workload Identity Blog.

Known Issues

To view the known issues for the 2.10 release of the Workload Identity feature, please refer to Known Issues