Configure

In this part, you will

• Create a custom cluster blueprint with the Kube Prometheus Stack and the Nginx ingress controller addon
• Create a Kubernetes secret
• Apply the custom cluster blueprint on your EKS cluster
• Update your DNS record

Step 1: Create Addons

In this step, you will

• Create namespaces where we will deploy our addons
• Create custom addons which we will use in our custom blueprint
• Provide the Kube Prometheus Stack addon with custom values so that we'll utilize no more than 10GB of storage, define a retention period of 7 days, define the dashboards we'll utilize, and define an ingress resource so we can access the Grafana dashboard

Create Namespaces

We will deploy our addons to a namespace called "prometheus" and "ingress-nginx"

• Click on Infrastructure -> Namespace
• Create a new namespace with the name "prometheus"
• Create a new namespace with the name "ingress-nginx"

Create Kube Prometheus Stack Addon

• Save the YAML provided below to a file called "kube-prometheus-stack-values.yaml" and update the fields appropriately

## We only care about values from k8s v1.15 and higher
#
kubeTargetVersionOverride: "1.15.12"

## Retain data for 7 days with max local storage of 10GB backed by a PVC
#
prometheus:
  prometheusSpec:
    retention: 7d
    retentionsize: 10G
  storageSpec:
    volumeClaimTemplate:
      spec:
        storage: 10Gi

## Do not use default dashboards. Specify the ones that are actually useful
## Add the required annotations for Ingress and Cert-Manager
#
grafana:
  defaultDashboardsEnabled: false
  adminPassword: "Password!23!"
  dashboards:
    default:
      kubernetes-cluster:
        gnetId: 12206
        datasource: Prometheus
      kubernetes-nodes:
        gnetId: 12133
        datasource: Prometheus
      kubernetes-pods:
        gnetId: 12128
        datasource: Prometheus
      kubernetes-node-exporter:
        gnetId: 12132
        datasource: Prometheus
      kubernetes-compute-namespace-pods:
        gnetId: 12117
        datasource: Prometheus
      kubernetes-api-server:
        gnetId: 12116
        datasource: Prometheus
      kubernetes-kubelet:
        gnetId: 12123
        datasource: Prometheus
      kubernetes-compute-cluster:
        gnetId: 12114
        datasource: Prometheus
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: default
          orgId: 1
          folder:
          type: file
          disableDeletion: true
          editable: false
          options:
            path: /var/lib/grafana/dashboards/default
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts:
       - grafana.dev.rafay-edge.net
    path: /
    tls:
    - secretName: grafana-dev-tls
      hosts:
       - grafana.dev.rafay-edge.net

• Click on Infrastructure -> Addons

• Click on Create New Addon from Catalog and select "kube-prometheus-stack"

• Enter "kube-prometheus-stack" for the Name

• Select the "prometheus" namespace from the dropdown

• Click on Create

• Provide a version (e.g. v65.1.0.1)

• Select the Version

• Upload the custom values file we just created.

Create Nginx Ingress Controller Addon

• Click on Infrastructure -> Addons

• Click on Create New Addon from Catalog and select "ingress-nginx"

• Enter "ingress-nginx" for the Name

• Select the "ingress-nginx" namespace from the dropdown

• Click on Create

• Provide a version (e.g. v4.10.4.1)

• Select the Version

Step 2: Create Blueprint

In this step, you will

• Create a custom cluster blueprint with a Kube Prometheus Stack and Nginx ingress controller addon.

• Select blueprints and create a new blueprint (e.g. kube-prometheus-stack)

• Click on "New version" (e.g. v1)

• Click on Configure Add-Ons

• Select the "kube-prometheus-stack" addon and "version" from the list of custom addons

• Select the "ingress-nginx" addon and "version" from the list of custom addons

• Add the "ingress-nginx" addon as a dependency to "kube-prometheus-stack"

• Click on Save Changes

• Deselect the Managed System Addon "Ingress Controller"

• Click on Save Changes

• Verify the blueprint has been created

Step 3: Create Kubernetes Secret

• Download the Cluster's Kubeconfig and set your context
• Navigate to the directory where you have the ".pem" file for the certificate and private key
• Create the Kubernetes secret in the "prometheus" namespace

kubectl create secret tls grafana-dev-tls --key="cert-key.pem" --cert="cert.pem" -n prometheus
secret/grafana-dev-tls created

Step 4: Apply Blueprint

Now, we are ready to apply the newly created, custom blueprint to our EKS cluster.

• Select Infrastructure -> Clusters
• Click on the gear icon on the far right of the EKS cluster
• Update blueprint and select the new blueprint and version

In a few minutes, all the k8s resources matching the custom cluster blueprint will become operational on the cluster. Notice that the cluster's blueprint name and version match what you created in the prior step.

Step 5: Verify Setup

Optionally, to verify if Prometheus is able to remote write to the AMP workspace

• Click on the EKS cluster
• Select Resources to view the integrated k8s dashboard
• Select "services" from the resource selector and filter by the "ingress-nginx" namespace
• Copy the External Endpoint for the Load Balancer as we will use this value for our DNS record.

Next Steps

You are now ready to move on to the next part of the recipe where you will access and visualize the time series data using Grafana.