Trivy

The two best known Kubernetes Security Benchmarks are CIS and NSA. As part of their work, they have released a set of recommendations for configuring Kubernetes clusters and workloads.

CIS (Center for Internet Security) benchmarks consists of secure configuration guidelines developed for Kubernetes. These include not only the running workloads but also the Kubernetes infrastructure set-up. CIS benchmarks are documented here.

NSA and CISA's guidance details recommendations to harden Kubernetes systems. Primary actions include the scanning of containers and Pods for vulnerabilities or misconfigurations, running containers and Pods with the least privileges possible, and using network separation, firewalls, strong authentication, and log auditing. NSA/CISA's Kubernetes Hardening Guidance report is available here.

This recipe describes how customers can standardize the deployment of the Trivy Operator, and generate CIS and NSA compliance reports.

What Will You Do

In this exercise,

• You will create the Trivy Operator add-on using Trivy's official Helm chart
• You will use the add-on in a custom cluster blueprint
• You will then apply this cluster blueprint to a managed cluster
• You will then query the Kubernetes benchmark reports generated by the Trivy Operator

Assumptions

• You have already provisioned or imported one or more Kubernetes clusters using the controller.

Step 1: Create the Trivy Operator add-on

1.1: Create Repository

• Log into the Web Console and navigate to your project as an Org admin or an Infrastructure Admin

• Under Integrations, select Repositories

• Create a New Repository

• Provide a name (e.g. trivy-helm-repo) and select type as Helm, click on Create

• Provide the URL for the Trivy helm repo, https://aquasecurity.github.io/helm-charts/

• Click on Save

1.2: Create Custom Catalog

• Navigate to the Catalog section, select Manage Catalogs

• Create a New Catalog

• Provide a name (e.g. trivy-catalog) and select the repo (e.g. trivy-helm-repo) created as part of the previous step

• Click on Sync Catalog

• Helm charts are now synced to the Custom Catalog, we will use the Trivy Operator Helm chart for this exercise

1.3: Create Namespace

You will now create a managed namespace from the controller for the Trivy Operator.

• Navigate to Infrastructure -> Namespaces

• Click on New Namespace

• Provide a name (e.g. trivy-system) and select type as Wizard

• Click Save

• Click "Save & Go To Placement"

• Select one of your clusters for placement

• Click "Save & Go To Publish"

• Click "Publish"

• Click "Exit"

1.4: Create custom Add-on

In this step, you will create a custom cluster add-on from the Catalog. The add-on will then be used in a custom cluster blueprint.

• Navigate to Add-Ons under the Infrastructure section

• Create a New Add-On, select the Create New Add-On from Catalog option

• Search for trivy-operator

• Click Create Add-On

• Provide a name (e.g. trivy-operator) and select the namespace (trivy-system) created as part of the previous step

• Click Create

• Enter v1 for the version name

• Click Save Changes

Step 2: Create Blueprint

In this step, you will create a custom cluster blueprint which contains the previously created add-on.

• Navigate to Blueprints under the Infrastructure section

• Click New Blueprint

• Provide a name (e.g. test-blueprint), select type as Custom Blueprint

• Click Save

• Enter v1 for the version name

• Select minimal for the base blueprint

• Click Configure Add-Ons

• Click the + symbol to add the trivy-operator add-on to the blueprint

• Click Save Changes

• Click Save Changes

Step 3: Apply Blueprint

In this step, you will apply the previously created blueprint to the cluster. Applying the blueprint will install the Trivy operator.

• Navigate to Infrastructure -> Clusters

• Click the gear icon on your cluster

• Select Update Blueprint

• Select the blueprint created as part of the previous step (e.g. test-blueprint) and v1 for the version

• Click Save and Publish

• Click Exit

Step 4: Query benchmark reports

4.1 Verify installation of Trivy Operator

• Navigate to Infrastructure -> Clusters

• Initiate a KubeCTL session through the web console

• Run the following command

kubectl get all -n trivy-system

The output should be similar to below.

4.2 Query reports

Once you have verified that everything is running correctly, you should be able to query the Kubernetes benchmark reports generated by the Trivy Operator.

kubectl get clustercompliancereport

• NSA report

kubectl get clustercompliancereport nsa -o yaml

• CIS report

kubectl get clustercompliancereport cis -o yaml

Step 5: Aggregating Results across fleet of clusters

With 10s or 100s of clusters organization wide, it is critical to have the means to centrally aggregate CIS benchmark reports to demonstrate "compliance" to the auditors. By default, the Trivy operator scans the cluster every 6 hours. Each scan results in an updated cluster compliance report that can be downloaded as described below.

You can use Rafay's Zero Trust Kubectl as the means to securely access their fleet of clusters to centrally aggregate the CIS Benchmark reports.

For example, an external pipeline can download the consolidated kubeconfig with all cluster contexts to retrieve the CIS scan results in seconds. See visual below for an illustrative example.

Cluster FleetRafay ControllerPipelineSplunkCluster FleetRafay ControllerPipelineSplunkloop[Every Cluster]loop[Every "x" Hours]Download consolidated kubeconfigRetrieve CIS scan report from each cluster using Rafay's zero trust kubectlRetrieve CIS scan reportStore CIS scan report

You can aggregate the reports in your organization's preferred CMDB such as Service Now or enterprise security platform such as Splunk. This ensures that the compliance team has visibility and access to every compliance report right from the birth of each cluster.

Recap

Congratulations! You have successfully created a custom cluster blueprint with Trivy Operator as a custom add-on. You can now use this blueprint on as many clusters as you require and query/generate and aggregate NSA and CIS compliance reports.