Setup

In this exercise, you will provision a Kafka Helm chart and test the autoscaling of Kafka with KEDA.

Important
This tutorial describes the steps using the Rafay Web Console. The entire workflow can also be fully automated and embedded into an automation pipeline.

Assumptions

You have already provisioned or imported a Kubernetes cluster into your Rafay Org and created a blueprint with KEDA.

Step 1: Create Namespace

• Login into the Web Console
• Navigate to Infrastructure -> Namespaces
• Create a new namespace, specify the name (e.g. kafka) and select type as Wizard
• In the placement section, select a cluster
• Click Save & Go to Publish
• Publish the namespace

Step 2: Create Kafka Add-on

• Navigate to Infrastructure -> Add-Ons

• Select New Add-On -> Create New Add-On from Catalog

• Search for kafka

• Select kafka from default-bitnami

• Select Create Add-On

• Enter a name for the add-on

• Specify the namespace (e.g. kafka and select the namespace created as part of the previous step

• Click Create

• Enter a version name

• Upload the following helm values

    persistence:
      enabled: false
    listeners:
      client:
        containerPort: 9092
        protocol: PLAINTEXT
        name: CLIENT
        sslClientAuth: ""
    ```  
- Click **Save Changes**

## Step 3: Update Blueprint  
- Navigate to **Infrastructure -> Blueprints**  
- Edit the previously created **KEDA** blueprint  
- Enter a version name  
- Click **Configure Add-Ons**  
- Select the previously created Kafka add-on  
- Click **Save Changes**  
- Click **Save Changes**

## Step 4: Apply Blueprint  
- Navigate to **Infrastructure -> Clusters**  
- Click the gear icon on your cluster and select **Update Blueprint**  
- Select the previously updated blueprint  
- Click **Save and Publish**

After a few seconds, the blueprint with the KEDA and Kafka add-ons will be published on the cluster.

## Step 5: Verify deployment  
- Navigate to **Infrastructure -> Clusters**  
- Click **KUBECTL** on your cluster  
- Type the following command

```bash
    kubectl get all -n kafka
    ```

## Step 6: Test Scaling  
### Create Kafka Consumer  
- Create a file named **consumer.yaml** with the following contents

```yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kafka-consumer
      namespace: kafka
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: kafka-consumer
      template:
        metadata:
          labels:
            app: kafka-consumer
        spec:
          containers:
            - name: consumer
              image: edenhill/kcat:1.7.1
              command: ["/bin/sh", "-c"]
              args:
                - kcat -C -b kafka.kafka.svc.cluster.local:9092 -G my-consumer-group test-topic
    ```

- Type the following command to create the resource

```bash
    kubectl apply -f consumer.yaml
    ```

- Type the following command to validate the resource was created

```bash
    kubectl get deployments -n kafka
    ```

### Create Kafka ScaledObject  
- Create a file named **scaledobject.yaml** with the following contents

```yaml
    apiVersion: keda.sh/v1alpha1
    kind: ScaledObject
    metadata:
      name: kafka-consumer-scaler
      namespace: kafka
    spec:
      scaleTargetRef:
        name: kafka-consumer
      minReplicaCount: 0
      maxReplicaCount: 5
      triggers:
        - type: kafka
          metadata:
            bootstrapServers: kafka-controller-headless.kafka.svc.cluster.local:9092
            topic: test-topic
            consumerGroup: my-consumer-group
            lagThreshold: "5"
            offsetResetPolicy: latest
    ```

- Type the following command to create the CRD

```bash
    kubectl apply -f scaledobject.yaml
    ```

- Type the following command to validate the resource was created

```bash
    kubectl get scaledobjects -n kafka
    ```

### Create Kafka Producer  
- Create a file named **producer.yaml** with the following contents

```yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kafka-producer
      namespace: kafka
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: kafka-producer
      template:
        metadata:
          labels:
            app: kafka-producer
        spec:
          containers:
            - name: producer
              image: edenhill/kcat:1.7.1
              command: ["/bin/sh", "-c"]
              args:
                - while true; do echo "test-message" | kcat -P -b kafka.kafka.svc.cluster.local:9092 -t test-topic; sleep 2; done
    ```

- Type the following command to create the CRD

```bash
    kubectl apply -f producer.yaml
    ```

- Type the following command to validate the resource was created

```bash
    kubectl get deployments -n kafka
    ```

The number of replicas for the producer can be scaled, this will increase the lag and cause KEDA to scale the consumer pods to keep up with the producers.