Overview

Container registries can be secured to prevent unauthorized entities from accessing images.

Kubernetes provides a feature called imagePullSecrets that allows pods to pull private docker images. To connect to a container registry, you have to add an ImagePullSecrets field to the configuration for a Kubernetes service account.

This is a type of Kubernetes secret that contains credential information. An imagePullSecrets is an authorization token, also known as a secret, that stores credentials for accessing a container registry.

Challenges

• It is operationally cumbersome to MANUALLY provision and manage Image Pull Secrets on a fleet of Kubernetes clusters
• It is an INSECURE practice to manually handle secrets
• It is BAD HYGIENE to have unused, dangling imagePullSecrets on clusters where the workload is no longer operational.

Our Solution

We have invested in developing deep integrations with Container Registry providers such as DockerHub (public, private), Amazon ECR, Google GCR, JFrog, Nexus etc.

• Project Admins can securely create, store, and manage imagePullSecrets for their private container registry repos.
• Developers can use these as "references/pointers" in their workloads.
• The Controller will securely deliver and automatically provision/deprovision the imagePullSecrets on the clusters where the workloads need to be deployed.

Benefits

• No manual handling of imagePullSecrets by developers or operations
• No need to embed imagePullSecrets in Kubernetes yaml files
• Secure (encrypted) delivery of imagePullSecrets to managed clusters
• Automated provisioning and deprovisioning of imagePullSecrets on clusters where workloads are deployed
• No dangling or orphaned imagePullSecrets on clusters
• Can rotate/refresh imagePullSecrets centrally and have it transparently apply on all managed clusters

Tested Integrations

Any Docker Compatible Registry should work. In addition, we test compatibility with the following providers with every release.

• System (Integrated Registry)
• Docker Hub (Public)
• Docker Hub (Private)
• AWS ECR
• Azure ACR
• GCP GCR
• Quay by RedHat
• Nexus by Sonatype
• JFrog Artifactory
• Microsoft MCR

Amazon ECR

Amazon's Elastic Container Registry is a fully-managed Docker container registry that allows developers to store, manage, and deploy Docker container images. Amazon ECR hosts container images in a highly available and scalable architecture.

Access to Amazon ECR requires client authentication. Successful authentication to an ECR registry provides an authorization token that is valid ONLY for 12 hours. Requiring authenticating every 12 hours ensures appropriate token rotation to protect against misuse.

Customer Challenge

Users that use Amazon ECR as their container registry but wish to pull container images on "Non Amazon" Kubernetes infrastructure will have to deal with constantly expiring authorization tokens.

The ECR Credential Helper

We have developed a Kubernetes CRD that will ensure that ECR authentication tokens are AUTOMATICALLY refreshed before they expire. This ensures that applications operating on non-Amazon Kubernetes infrastructure can seamlessly access container images from Amazon ECR on an ongoing basis.

Manage Registry Integration

Project Admins can configure Registries in the Web Console.

• Under Integrations, Select Registry

• Create Registry

• Select Registry type

• Enter access credentials

The credentials are encrypted before they are stored. See Security section for additional details on Key Management.

Registry Creation Success

On successful registry creation, users can view the list in the Container Registries page and perform a few actions if required.

• Display Annotation icon to view the annotation attached to this container registry. Use the copy icon to use the annotation for deploying a workload.
• Validate icon to check the registry valid/invalid status
• Edit icon to make any changes to the existing registries
• Delete icon to remove the unnecessary registries from the system

Use Registry Integration

Once a registry integration has been created by a Project Admin, it can be referenced and used in workloads operating in the Project.

Workload Wizard

For the wizard-based workloads,

• Select the registry integration from the dropdown
• Enter the repository and tag

During the deployment of the workload, the controller will automatically provision the imagePullSecrets to the targeted clusters.

Annotations for Yaml or Helm

For Kubernetes YAML and Helm based workloads, we provide "annotations" that can be quickly added to an existing k8s yaml or Helm values.yaml. Annotation that has to be specified in the yaml file can be retrieved as below.

• Under Integrations, Select Registries
• Select the Registry that you have configured previously
• Under Actions, Select Display Annotation to copy the annotation.

  annotations:
    rafay.dev/registry-key1:value1
    rafay.dev/registry-key2:value2

The reason for different keys is to ensure support for situations with pod specs with multiple containers where the registries from which images need to be pulled are different.

As a result, the registry annotations need to contain references for multiple registry instances.

Here is an example of a k8s Deployment YAML that will pull the images from Amazon ECR.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: km-nginx2
  annotations:
   rafay.dev/registry-ecr: km-ecr
spec:
  selector:
    matchLabels:
      app: km-nginx2
  replicas: 1
  template:
    metadata:
      labels:
        app: km-nginx2
    spec:
      containers:
      - name: km-nginx2
        image: 679196758854.dkr.ecr.us-east-1.amazonaws.com/km-nginx:1.17
        ports:
        - containerPort: 80

System Registry

An integrated system container registry is available for non-production use (e.g. testing, demos etc.). This registry is based on Docker Registry v2 and also hosts community template container images.

Follow the instructions on how to authenticate and use the system registry.