Configuration Examples for EKS System Template - Rafay Product Documentation

Config Examples

Configuration Examples for EKS System Template

This document provides configuration snippets and usage patterns for customizing your Amazon EKS clusters using the EKS System Template in Rafay.

Control Plane Subnet IDs

To specify the subnets for your EKS control plane, use the Control Plane Subnet IDs input parameter in your system template:

["subnet-070c1e8xxxxx", "subnet-02c612dd9yyyyy", "subnet-0b3bea6zzzzz", "subnet-0700b95vvvvvvv"]

Managed Node Groups Configuration

To create managed node groups with custom configurations — such as instance types, AMI settings, subnet placement, and bootstrap arguments — use the EKS Managed Node Group input variable in your system template. This allows you to define multiple managed node groups with different configurations.

Example 1: Basic Managed Node Group Configuration

{
  "managed-ng-1": {
    "ami_type": "AL2_x86_64",
    "instance_types": ["t3.medium"],
    "cluster_version": "1.31",
    "min_size": 1,
    "max_size": 5,
    "desired_size": 2
  },
  "managed-ng-cami": {
    "ami_type": "AL2_x86_64",
    "instance_types": ["t3.medium"],
    "min_size": 1,
    "max_size": 5,
    "desired_size": 2,
    "ami_id": "ami-00ae057af58f63a80",
    "enable_bootstrap_user_data": true,
    "bootstrap_extra_args": "--container-runtime containerd --kubelet-extra-args '--max-pods=46' '--max-open-files=1000000' '--container-log-max-files=10'",
    "subnet_ids": [
      "subnet-0b3bea60007a1e322",
      "subnet-0700b95caf66b04da"
    ]
  }
}

Example 2: Managed Node Group with Custom Encrypted EBS Volumes

{
  "mg-node-group-01": {
    "ami_type": "AL2_x86_64",
    "instance_types": ["t3.medium"],
    "subnet_ids": ["subnet-09f11fbddxxxx", "subnet-0dc7b0289yyyyy", "subnet-01f66d22bzzz"],
    "min_size": 0,
    "max_size": 2,
    "desired_size": 2,
    "block_device_mappings": [{
      "device_name": "/dev/xvda",
      "ebs": {
        "encrypted": true,
        "kms_key_id": "arn:aws:kms:us-west-2:xxxxx:key/xxxx-xxxxx",
        "volume_size": 100,
        "volume_type": "gp2"
      }
    }]
  }
}

Cluster Additional Security Group IDs

To attach additional security groups to the EKS control plane, use the Cluster Additional Security Group IDs field in your system template:

["sg-07f01xxxxxxxxx"]

Node IAM Role ARN

To assign a custom IAM role to the EKS worker nodes (managed node groups), use the Node IAM Role ARN field. This is useful when you want nodes to assume a specific IAM role with fine-grained permissions:

"arn:aws:iam::xxxxxxxx:role/lan-AmazonEKSNodeRole"

Cluster Addons Configuration

The Cluster Addons field allows you to configure and manage EKS-managed add-ons such as CoreDNS, kube-proxy, VPC CNI, and CSI drivers directly within the EKS System Template. You can also attach IAM roles to service accounts (using Pod Identity Associations) and provide custom configurations or versions for specific add-ons.

Example: Cluster Addons Configuration

{
  "coredns": {
    "name": "coredns",
    "pod_identity_association": []
  },
  "kube-proxy": {
    "name": "kube-proxy",
    "pod_identity_association": []
  },
  "eks-pod-identity-agent": {
    "name": "eks-pod-identity-agent",
    "pod_identity_association": []
  },
  "aws-ebs-csi-driver": {
    "name": "aws-ebs-csi-driver",
    "addon_version": "v1.38.1-eksbuild.2",
    "configuration_values": <<JSON
    {
      "defaultStorageClass": {
        "enabled": true
      }
    }
    JSON
  },
  "vpc-cni": {
    "name": "vpc-cni",
    "pod_identity_association": [
      {
        "role_arn": "arn:aws:iam::xxxxxxx:role/kalyan-vpc-cni-pia-role",
        "service_account": "aws-node"
      }
    ]
  }
}

Node Security Group Additional Rules

Use the Node Security Group Additional Rules field to define custom ingress or egress rules for the node security group. This is useful when you need to open specific traffic flows between pods, the control plane, or other network components.

Example: Allow Pod-to-Control Plane Traffic

{
  "allow-traffic-from-pods-to-control-plane" = {
    description                   = "allow traffic from pods to control plane"
    protocol                      = "-1"
    from_port                     = 0
    to_port                       = 0
    type                          = "ingress"
    source_cluster_security_group = true
  }
}

Key Fields:

type: Direction of the rule (ingress or egress)
protocol: Protocol used (-1 means all protocols)
from_port / to_port: Port range (0-0 for all ports)
source_cluster_security_group: true allows traffic from the cluster’s security group

Pod Identity Association

To associate IAM policies with Kubernetes service accounts via Pod Identity Associations, use the following configuration format.

Example 1: Pod Identity Association for AWS Load Balancer Controller

{
  "aws-load-balancer-controller" = {
    additional_policy_arns = {
      "aws-load-balancer-controller-policy" = "arn:aws:iam::xxxxxx:policy/lb-aws"
    },
    associations = {
      aws-load-balancer-controller = {
        namespace       = "kube-system"
        service_account = "aws-load-balancer-controller"
      }
    }
  }
}

In the above example, the key aws-load-balancer-controller-policy refers to the IAM Policy ARN:

"arn:aws:iam::xxxxxx:policy/lb-aws"

This policy ensures that the AWS Load Balancer Controller has the necessary IAM permissions via pod identity.

Example 2: Pod Identity Association for Karpenter

{
  "karpenter-sa" = {
    additional_policy_arns = {
      "karpenter-policy" = "arn:aws:iam::xxxxxxx:policy/demo-karpenter-policy"
    },
    associations = {
      karpenter-sa = {
        namespace       = "karpenter"
        service_account = "karpenter"
      }
    }
  }
}

Node Repair Support for EKS Managed Nodegroups

Enabling Node Repair for EKS managed nodegroups provides the following benefits:

Automatically replaces or reboots unhealthy nodes to maintain cluster health
Improves high availability and reduces manual intervention
Ensures consistent performance in production environments

To enable AWS-native Node Repair for EKS managed nodegroups, add the following configuration in the node group spec:

"node_repair_config": {
  "enabled": true
}

Example: Node Group Configuration

"default-node-group": {
  "ami_type": "AL2_x86_64",
  "instance_types": ["t3.medium"],
  "min_size": 1,
  "max_size": 5,
  "desired_size": 2,
  "node_repair_config": {
    "enabled": true
  }
}

Example: Cluster Addons Configuration

"addons": {
  "coredns": {},
  "eks-pod-identity-agent": {},
  "kube-proxy": {},
  "vpc-cni": {},
  "eks-node-monitoring-agent": {
    "most_recent": true
  }
}