KOP EKS Clusters - CNI Providers - Rafay Product Documentation

CNI Providers

CNI Providers for Kubernetes

Networking is a critical component that ensures smooth communication between containers, services, and applications. To make this process simpler and more flexible, Kubernetes clusters support various Container Network Interface (CNI) providers. These providers enable the configuration of networking plugins that help manage networking resources and communication across containers.

EKS clusters, in particular, support three prominent CNI providers: AWS VPC CNI, Cilium, and Calico. Each of these providers has its own unique set of features, allowing users to select the best networking solution for their specific needs.

💡 Tip: Users can add any CNI when creating an addon; however, Cilium, Calico, and AWS VPC are thoroughly qualified and tested to work seamlessly.

Users can choose and customize their preferred CNI provider's configuration through multiple interfaces, including UI, Terraform, RCTL, and API, ensuring seamless integration with their Kubernetes environments.

AWS VPC

AWS uses Amazon VPC Container Network Interface (CNI) plugin for the pod networking in EKS clusters provisioned by the controller. AWS CNI is the default CNI Provider. AWS VPC ensures that CNI plugin daemons are scheduled on all nodes in a cluster and adds elastic network interface (ENI) resource tagging.

Calico

Calico is a popular networking solution used to interconnect virtual machines or Linux containers with the help of virtual routers. Calico provides a Cluster Network Interface (CNI) plugin that can be used for integration with Kubernetes. Users who prefer flexible IP address management capabilities can select the Calico CNI provider. On selecting Calico, all the pods related to Calico gets installed.

Cilium

Cilium uses eBPF (Extended Berkeley Packet Filter) technology to provide high-performance networking, enhanced security, and deep network visibility. It is especially beneficial for users looking for advanced load balancing, security policies, and observability in cloud-native environments.

CNI Providers in Controller

During EKS Cluster provisioning, user must select a CNI Provider from the CNI Providers drop-down available in Advance Settings. If none of the CNI is selected, AWS is set as the default CNI provider.

Why Is This Useful?

The introduction of multiple CNI providers allows users to choose a networking solution that best fits their specific requirements, whether it’s for performance, security, or flexibility. For instance, users who prioritize high security and advanced network observability might prefer Cilium, while those requiring simplified IP address management could lean toward Calico.

CNI Providers through CLI

Users can use the cluster config yaml file to provision the cluster with Calico CNI. Below is an illustrative example of a cluster config file with the CNI Provider Calico

kind: Cluster
metadata:
  name: demo-ekscluster
  project: project1
spec:
  blueprint: bp-with-co-addon
  blueprintversion: v1
  cloudprovider: demo-provider
  clusterConfig: null
  cniprovider: Calico-v3.19.1
  proxyconfig: {}
  type: eks
---
apiVersion: rafay.io/v1alpha5
kind: ClusterConfig
metadata:
  name: demo-ekscluster
  region: us-west-1
  version: "1.19"
nodeGroups:
- amiFamily: AmazonLinux2
  desiredCapacity: 2
  iam:
    withAddonPolicies:
      autoScaler: true
      imageBuilder: true
  instanceType: t3.xlarge
  maxSize: 2
  minSize: 2
  name: ng-2220fc4d
  volumeSize: 80
  volumeType: gp3
vpc:
  cidr: 192.168.0.0/16
  clusterEndpoints:
    privateAccess: true
    publicAccess: true
  nat:
    gateway: Single

Use the below command in the terminal to apply the changes

./rctl apply -f <filename>

Important

CNI Customization

CNI Customization

Previously, users were limited to selecting Container Network Interfaces (CNIs) from a predefined CNI list, which did not allow for any customization of CNI values. With the latest enhancement, users now have the flexibility to customize Calico and Cilium CNI values by leveraging addons and blueprints.

⚠️ Important Note: Required Labels for CNI Add-ons

When creating a new add-on, it is mandatory to include the following labels to ensure proper configuration:

  1. key: rafay.type and value: cni
  2. **key: rafay.cni.name**and one of the following values:
    • value: cilium
    • value: calico

Calico CNI Customization

Users can upload the Calico Helm chart and value files of the required version through any interface—UI, RCTL, API, or Terraform. This method provides flexibility to customize the CNI values based on specific environment requirements. It’s important to note that adding labels is mandatory when creating a CNI add-on, as these labels are essential for proper configuration.

Here is an example of creating an add-on with the Calico CNI. Similarly, add-ons can be created for Cilium CNIs

Step 1: Create a Namespace

Step 2: Create a Add-On

Example Labels for This Case:

⚠️ Important Note: Calico CNI Installation Configuration

It is mandatory to update the values file with the following configuration for Calico CNI: installation: kubernetesProvider: EKS cni: type: Calico calicoNetwork: bgp: Disabled

Here is the newly created add-on with the Calico CNI Helm charts and value files.

Step 3: Create a Blueprint

The blueprint is now created with the calico CNI add-on

Step 4: Create a Cluster

Once the cluster is created, run the command kubectl get pods -A to view the Calico CNI running status, as shown in the image below.

⚠️ Key Note

When a CNI-based blueprint is added, the cluster is initially provisioned using the default AWS CNI. Once the cluster provisioning is complete, including the deployment of all resources, the AWS CNI is automatically removed. Only the CNI specified in the blueprint will remain deployed and active within the cluster.

Cilium CNI Customization

Similar to Calico, users can add a CNI-based blueprint with Cilium CNI, but only on Day 2. To achieve this:

Day-2 Operations

In addition, users now have the option to modify Calico and Cilium CNI values as part of Day 2 operations. To do this, create a new version of the addon by uploading an updated Helm chart and value files or by editing the existing value file. Below is an example of cilium values file where the ui automount value is modified from true to false. Click Update to apply the changes.

Update the blueprint with the new addon version

Now, update the cluster blueprint to deploy the latest or modified Cilium CNI to the cluster. This enhancement provides added flexibility for ongoing network configuration adjustments that were previously unavailable.

This will deploy the updated version of the blueprint to the cluster, and the deployment status is shown below.

Migration to Blueprint-Based CNI Configuration

🚀 For older upstream clusters currently using Cilium or Calico without a blueprint-based setup, migrating to a blueprint-based configuration for upgrades or other Day 2 operations requires attention to the following:

Key Steps for Migration

To ensure a seamless migration, follow these highlighted steps: