# Backup and Restore using Velero with S3-Compatible Storage

This guide explains how to configure **Velero** with **any S3-Compatible Storage** as the backend storage for performing backup and restore operations in an **air-gapped controller** environment.

Unlike public S3 services, this setup assumes **S3-compatible storage is hosted on a separate Linux server** within the same **private network**, ensuring data locality and offline functionality.

---

## Prerequisites

Ensure the following are available before proceeding:

- An S3-compatible storage server accessible over the private network
- A dedicated bucket in the S3-compatible storage for Velero backups
- S3 access credentials (Access Key and Secret Key)
- S3 API endpoint: `http://<S3_SERVER_IP>:9000`
- S3 Console UI: `http://<S3_SERVER_IP>:9001` (if available)
- Network connectivity between the Kubernetes cluster and the S3-compatible storage server

---

## Backup Implementation Procedure

Follow the steps below to enable and schedule backup jobs using Velero with any S3-compatible storage.

> **Note:** If backups are already enabled and scheduled, this section can be skipped.

### Step 1: Create an S3-Compatible Storage Bucket

> Skip this step if a backup bucket already exists.

Use your S3-compatible storage client (e.g., MinIO Client `mc`, AWS CLI, etc.) or the console UI to create the bucket:

```
# Using MinIO Client (example)
mc alias set mys3 http://<S3_SERVER_IP>:9000 <ACCESS_KEY> <SECRET_KEY>
mc mb mys3/<YOUR_BUCKET_NAME>

# Or access the Console UI at (if available):
http://<S3_SERVER_IP>:9001
```

### Step 2: Verify S3-Compatible Storage Connectivity

```
# List buckets to verify access (example using MinIO Client)
mc ls mys3/
```

### Step 3: Update the `backup_restore` Section in config.yaml

Update the config.yaml file as shown below:

```
backup_restore:
  enabled: true
  restore: false
  schedule: "0 0 * * *"  # Use cron syntax for scheduling (e.g., every day at midnight)
  bucketName: "rafay-core-backup"
  retentionPeriod: "168h0m0s"  # Retain backups for 7 days
  resticEnable: true           # Enable pod volume backups
  snapshotsEnabled: true       # Enable volume snapshots

# External Blob Storage (S3-Compatible) credentials (base64 encoded)
  externalBlobStorage:
    username: ""  # Base64-encoded Access Key
    password: ""  # Base64-encoded Secret Key
    endpoint: ""  # e.g., http://<S3_SERVER_IP>:9000
```

### Step 4: Enable Backup Support with Velero

Run the following command to install controller dependencies with Velero enabled:

```
sudo radm dependency --config config.yaml
```

> **Note:** For S3-compatible storage endpoints using self-signed certificates, you may need to patch the backup storage location to skip TLS verification. Run the following command:
>
>```
kubectl patch backupstoragelocation default -n velero --type merge -p '{"spec":{"config":{"insecureSkipTLSVerify":"true"}}}'
```

### Step 5: Verify Backup Job Execution

Use the command below to confirm scheduled backups are being created:

```
kubectl get backups -n velero
```

**_Sample output_**

```
NAMESPACE   NAME                                      AGE
velero      velero-rafay-core-backup-20240403100012   22h
velero      velero-rafay-core-backup-20240403110012   21h
velero      velero-rafay-core-backup-20240403120013   20h
velero      velero-rafay-core-backup-20240403130013   19h
```

### Step 6: Check Backup Status

Verify backup status and progress:

```
kubectl describe backup -n velero velero-rafay-core-backup-20240403130013
```

**_Sample output (excerpt)_**

```
Status:
  Completion Timestamp:  2024-04-04T05:34:10Z
  Expiration:            2024-04-11T05:33:37Z
  Phase:                 Completed
  Progress:
    Items Backed Up:           9037
    Total Items:               9037
  Volume Snapshots Attempted:  13
  Volume Snapshots Completed:  13
```

---

## Restore Procedure

Follow the steps below to restore the controller from an existing backup.

### Step 1: Prepare the Controller Package

- Download and extract the controller tarball
- Ensure the `config.yaml` file is aligned with the previous (backup) configuration
- Update the `backup_restore` section in `config.yaml` as follows:

```
backup_restore:
  enabled: true
  restore: true
  bucketName: "rafay-core-backup"
  restoreFolderName: "velero-rafay-core-backup-20250519130015"  # Update with latest backup folder name
  resticEnable: true
  snapshotsEnabled: true

> ⚠️ Ensure `restoreFolderName` matches the backup name to be restored.

### Step 2: Initialize the Controller

```
sudo radm init --config config.yaml
```

### Step 3: Deploy the Controller with Restore Enabled

```
sudo radm dependency --config config.yaml
sudo radm application --config config.yaml
```

After completion, the controller will be restored using the latest Velero backup from the S3-compatible storage.

---