Backup and Restore with AWS S3 - Rafay Product Documentation

Backup and Restore Guide for Air-Gapped Controllers

In highly secure environments where air-gapped Rafay controllers are deployed, it is critical to implement a reliable backup and restore strategy. This guide outlines how to:

Architecture Overview

The following diagram illustrates the backup and restore workflow for air-gapped Rafay controllers in a disaster recovery scenario:

Prerequisites

When to Configure

You can set up this configuration at:

Note: For Day-2 configuration, make sure to run the following command to install the required Velero charts on the controller:

sudo radm dependency --config config.yaml

Backup Configuration Steps

We use Velero for backup and restore of the air-gapped controller. To allow Velero to authenticate with the external S3-compatible object store, you need to securely create and store AWS credentials as a Kubernetes secret.

1. Create an S3 Bucket

If not already available, create an S3 bucket to store the backups:

aws s3api create-bucket \
  --bucket <YOUR_BUCKET> \
  --region <REGION> \
  --create-bucket-configuration LocationConstraint=<REGION>

Note: Skip this step if you're using an existing S3 bucket.

2. Create IAM Policy and Attach to User

Prepare a policy with permissions required for Velero to perform backup and restore operations:

BUCKET=<YOUR_BUCKET>
cat > velero-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [\
    {\
      "Effect": "Allow",\
      "Action": [\
        "ec2:DescribeVolumes",\
        "ec2:CreateVolume",\
        "ec2:CreateSnapshot",\
        "ec2:DeleteSnapshot",\
        "ec2:CreateTags",\
        "ec2:DescribeSnapshots"\
      ],\
      "Resource": "*"\
    },\
    {\
      "Effect": "Allow",\
      "Action": [\
        "s3:GetObject",\
        "s3:PutObject",\
        "s3:DeleteObject",\
        "s3:ListBucket",\
        "s3:AbortMultipartUpload",\
        "s3:ListMultipartUploadParts"\
      ],\
      "Resource": [\
        "arn:aws:s3:::${BUCKET}",\
        "arn:aws:s3:::${BUCKET}/*"\
      ]\
    }\
  ]
}
EOF

Attach the policy to your IAM user:

aws iam put-user-policy \
  --user-name <USERNAME> \
  --policy-name velero-access \
  --policy-document file://velero-policy.json

3. Create Secret for S3 Credentials

a. Generate the credentials file

On the node where you are preparing the controller, create a file named cloud with the following content:

cat <<EOF > ~/cloud
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
EOF

Replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with valid values for your S3 object store.

b. Create the Kubernetes Secret

Use the generated file to create the secret in the velero namespace:

kubectl create ns velero
kubectl create secret generic velero \
  --namespace velero \
  --from-file ~/cloud

Info: The cloud file is not used directly by Velero. Instead, the contents are packaged into a Kubernetes secret that Velero uses to authenticate with your object store. This approach avoids embedding sensitive credentials in config.yaml, reducing security risks and aligning with best practices.

4. Update Controller Configuration

Edit the config.yaml file with the following section under backup_restore:

backup_restore:
  enabled: true
  restore: false
  schedule: "0 0 * * *"           # Daily at midnight
  bucketName: "<YOUR_BUCKET>"
  retentionPeriod: "168h0m0s"     # 7-day retention
  backupFolderName: "controller-backups"

Note: Minimum interval supported is 45 minutes to ensure each backup completes before the next one starts.

Note: Only the first backup is a full backup. Others are incremental. If the full backup expires, the incremental backups depending on it become unusable.

This enables automated backup scheduling to your designated S3 bucket.

5. Run Installation Commands

# Install dependencies for Rafay controller
sudo radm dependency --config config.yaml

# Install controller application
sudo radm application --config config.yaml

Once these steps are completed, your controller will automatically back up to the S3 object store as per the defined schedule.

Restore Procedure

Note: Perform the following steps on the newly provisioned controller node, or on a passive controller if you already have one available. Ensure you have access to the required backup files and credentials before proceeding.

Follow these steps to restore your Rafay air-gapped controller from backup:

  1. Prepare S3 Credentials

On the controller node, create a file named cloud with your S3-compatible object store credentials:

cat <<EOF > ~/cloud
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
EOF

Replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your actual credentials.

  1. Create Kubernetes Secret for Velero

Use the credentials file to create a Kubernetes secret in the velero namespace:

kubectl create ns velero
kubectl create secret generic velero \
     --namespace velero \
     --from-file ~/cloud
  1. Download and Extract Controller Package

Obtain the Rafay controller tarball and extract it on the controller node.

  1. Update config.yaml for Restore

Edit the config.yaml file and update the backup_restore section as follows:

backup_restore:
     enabled: true
     restore: true
     schedule: "0 0 * * *"           # Daily at midnight (can be adjusted)
     bucketName: "<YOUR_BUCKET>"     # Name of your S3 bucket
     retentionPeriod: "168h0m0s"     # 7-day retention
     backupFolderName: "<LATEST_BACKUP_FOLDER>"  # Name of the latest backup folder

Ensure restore: true and provide the correct bucket and backup folder names.

  1. Install Dependencies and Restore Controller

Run the following commands to install dependencies and restore the controller application:

sudo radm dependency --config config.yaml
sudo radm application --config config.yaml
  1. Post-Restore Steps

Once the restore is complete: