Get Started with Auto Mode for Amazon EKS with Rafay

This is Part 3 in our series on Amazon EKS Auto Mode. In the previous posts, we explored:

1. Part 1: An Introduction: Learn the core concepts and benefits of EKS Auto Mode.
2. Part 2: Considerations: Understand the key considerations before Configuring EKS Auto Mode.

In this post, we will dive into the steps required to build and manage an Amazon EKS cluster with Auto Mode template using the Rafay Platform. This exercise is specifically well suited for platform teams interested in providing their end users with a controlled self-service experience with centralized governance.

Overview

In this exercise, you will use Environment Manager to create and manage an Amazon EKS cluster operating in Auto Mode. These resources will be controlled as a single environment in the Rafay Platform Environment Manager allows Development and DevOps teams to create environments while giving central Ops and SRE teams control over security, cost, and governance. The EKS cluster is defined using OpenTofu based Infrastructure as Code that is stored in the organization's Git repository (i.e. GitOps)

Setup Environment Template
Provision & Use Environment
Set Git Repo Containing the IaC Code

1. Setup Env Manager Resources (Config Context, Resource Template, Environment Template)
2. Create Environment Template
3. Share Environment Template to User's Project
4. Launch Environment
5. Use Environment

Objective

Platform teams at many organizations may wish to provide their end users (e.g. developers) with a self-service experience to provision and use EKS clusters in Auto Mode. But, they also do not want this to become a Wild West situation. For example, it does not make sense to expose every option to the developers since it will not only overwhelm them, it will also result in significant security/governance issues.

In this guide, we will create a self-service workflow to:

• Provision an Amazon EKS cluster with Auto Mode.
• Limit/Control what the user needs to configure and abstract out the rest.
• Automate daily shutdown (at 9pm) and auto restart (at 9am) for cost control.
• Deploy the Rafay Operator into the EKS cluster for centralized management.

Design and Architecture

We will wrap the standard Infrastructure as Code (IaC) provided by AWS using an Environment Template and provide our users with a self-service experience to configure and provision EKS Auto Mode clusters. In our design, the environment template is created using the Rafay platform and is persisted in the Git repository alongside the IaC. The Rafay platform is able to connect to both the customer's Git repository and their AWS account using a Rafay Agent that is deployed in the customer's infrastructure.

The declarative spec for the environment template itself has several components as shown in the image below. Notice that there are two resource templates attached to it. One of them is to manage the lifecycle of the EKS cluster and the second one is for the Rafay Kubernetes Operator.

Follow the steps described below to configure and provision Amazon EKS Auto Mode clusters using Rafay. Alternatively, you can also watch a video of the end-to-end experience of the same steps.

Provision & Operate Amazon EKS Auto Mode Clusters with Rafay | Hands-Free Kubernetes - YouTube

Provision & Operate Amazon EKS Auto Mode Clusters with Rafay | Hands-Free Kubernetes

Assumptions

Before proceeding, ensure you have the following:

1. AWS Account Access: with the necessary permissions to create resources such as EKS, VPC etc.
2. Git Repository: We will manage version controlled OpenTofu IaC code for EKS Auto Mode and the Rafay operator.
3. Rafay Platform: We will manage environments in the Rafay Org

Note

You can sign up for a free Rafay Org to try this out.

Step-by-Step Guide

Follow the steps described below

Step 1: Create an Agent

1. Navigate to the Agent Creation Page:

   • Go to Gitops > Gitops Agent > New Agent in the Rafay Controller.

2. Create the Agent:

   • Provide an Agent Name and select Docker Container as the deployment type.

3. Obtain Docker Compose and Config File:

   • Once the agent is created, the controller will generate a docker-compose.yaml and relayConfigData file. Use these files to deploy the agent.

4. Deploy the Agent:

• Deploy the agent on a machine within your network using Docker Compose:

  sudo docker-compose -f <docker-compose-file>.yaml up -d
  

5. Verify the Health:
   • Check the controller to ensure the agent’s status is Healthy. This indicates the agent is ready to interact with the Git repository and the Rafay Controller.

Step 2: Set Up a GitHub Repo Integration

1. Navigate to the Integration Page:
   • Go to Integrations > Repositories > New Repository on the Rafay platform.
2. Link Your GitHub Repository:
   • Add your private GitHub repository containing the Terraform code for:
     • EKS Cluster Auto Mode
     • Rafay Import Cluster

Note: The IaC code for EKS Auto Mode is available in this test repository.

Example: For this demo, I referenced the IaC code for EKS Auto Mode from the AWS Terraform EKS Module repository and customized it a bit to meet my requirements.

Step 3: Create Environment Resources

Configure Contexts

The config context encapsulates and manages credentials that will be used by the Rafay Agent to securely interact with your Rafay Org and your AWS account. The config context can be reused in other environment templates as well.

1. Rafay Config Context: Add the API key and endpoint for Rafay.
2. AWS Config Context: Add AWS credentials for creating the EKS cluster.

Example: Config Contexts Overview

Example: Rafay Config Context

Example: AWS Config Context

Create Resource Templates

A resource template is equivalent to a reusable "Lego Block". Platform teams can assemble complex environments using many of these resource templates (just like they would using Lego blocks).

1. EKS Lifecycle Template:
   • Navigate to Environments > Resource Templates > Create New Resource Template.
     • Provide a name and description, then click Create.
     • Add a new version:
     • Specify the version name and set the provider to opentofu.
     • Reference the Git repository, branch, and folder path containing the Terraform code.

• Attach the AWS credential config context.

2. Rafay Import Template (Apply Rafay Operator):
   • Create the second resource template for the import cluster:
     • Follow the same steps as the EKS Cluster template creation.
     • Reference the folder containing Terraform code for Rafay import.
     • Configure input variables such as:
       • blueprint_version
       • kubernetes_provider
       • project_name
       • provision_environment

Note: Attach the agent to drive the workflow. You can attach the agent at any of two places:

• At any resource template that is part of the environment.
• At the environment template.

Important

If you want to automatically fetch variables from the IaC code, you must have an agent mapped to the respective resource template.

Step 4: Create the Environment Template

In this step, as an admin, you will assemble various resource templates together, setup dependencies, configure TTL/schedules and decide what configuration parameters you want to expose to the end user.

1. Combine Resource Templates:
   • Add the EKS Cluster and Rafay Import Cluster resource templates.
2. Define Dependencies:
   • Ensure the Rafay import step depends on the successful creation of the EKS cluster.

3. Expose Input Variables

To enhance the user experience, expose input variables with user-friendly names by using selectors. This ensures that the actual input variables are displayed in an intuitive format for end users.

• Add Input Variables:
  • Cluster Name
  • Region
  • Kubernetes Version
• Apply Selectors for Aliasing:
  • Use selectors to alias actual input variables with user-friendly names. This makes it easier for users to understand and input values.

4. Automate Schedules:
   • Configure daily shutdown at 9 PM IST and recreation at 9 AM IST for cost optimization.

End User: Deploying the Environment

In this step, ideally login as an end user with the role "Env Template User" to see the user experience you can deliver for end users.

1. Launch the Environment Template:

   • Navigate to the environment template and initiate deployment.

2. Provide Input Variables:

   • Fill in deployment-specific variables such as:
     • Cluster Name
     • Region
     • Blueprint Name
     • Kubernetes Version

3. Validate Deployment:

   • Verify that:
   • The EKS cluster is created with Auto Mode.
   • The cluster is successfully imported into Rafay.

Example: Import Cluster

Conclusion

In this blog and supporting video, we saw how easy it is for organizations to provision and operate Amazon EKS Auto Mode clusters using the Rafay Platform. In a follow-on blog, we will cover the following topic:

How does a Kubernetes Management offering such as Rafay add value for organizations standardized on Amazon EKS Clusters in Auto Mode?