# Air-Gapped Controller Troubleshooting

This guide covers common issues and solutions for air-gapped Rafay controllers.

## Controller Issues

### TLS & Certificate Issues

**Problem**: UI access fails with certificate errors or container image pulls fail due to TLS validation.

**Solution**: Replace/renew expired TLS certificates using the following steps:

1. **Prepare Certificate Files**  
2. Generate new TLS certificate and private key for wildcard controller domain (e.g., `*.controller.example.com`)  
3. Save certificate chain as `tls.crt`  
4. Save private key as `tls.key`

5. **Backup Existing TLS Secrets**

```
   kubectl get secret admin-ingress-certs -n istio-system -o yaml > admin-ingress-certs.yaml
   kubectl get secret rafay-container-registry-tls-secret-opaque -n rafay-core -o yaml > rafay-container-registry-tls-secret-opaque.yaml
   ```

6. **Update TLS Secrets**

```
   kubectl create secret generic admin-ingress-certs \
      --from-file=tls.crt=tls.crt \
      --from-file=tls.key=tls.key \
      -n istio-system -o yaml --dry-run=client | kubectl apply -f -

kubectl create secret generic rafay-container-registry-tls-secret-opaque \
      --from-file=tls.crt=tls.crt \
      --from-file=tls.key=tls.key \
      -n rafay-core -o yaml --dry-run=client | kubectl apply -f -
   ```

7. **Restart Affected Deployments**

```
   kubectl rollout restart deployment/istio-ingressgateway -n istio-system
   kubectl rollout restart deployment/admin-api -n rafay-core
   ```

Note: Check the validity of the controller console URL before and after the certificate replacement to ensure the new certificate is applied correctly and trusted by the browser.

---

### Controller UI Accessibility Issues

**Problem**: The controller console UI is inaccessible or unresponsive.

**Troubleshooting Steps**:

1. **SSH into the controller master node** and check if all controller pods are running:

```
   kubectl get pods -A
   ```

2. **Identify problematic pods**: Look for pods in `CrashLoopBackOff`, `Error`, or `Pending` state.

3. **Describe the problematic pod** to get more details about the issue:

```
   kubectl describe pod <pod-name> -n <namespace>
   ```

4. **Check pod logs** for error messages:

```
   kubectl logs <pod-name> -n <namespace>
   ```

5. **Scale down and scale up the deployment** to restart the affected pods:

```
   # Scale down the deployment
   kubectl scale deployment <deployment-name> -n <namespace> --replicas=0

# Wait a few seconds, then scale back up
   kubectl scale deployment <deployment-name> -n <namespace> --replicas=1
   ```

6. **Verify all pods are running** after the restart:

```
   kubectl get pods -A
   ```

7. **Try accessing the controller UI** again once all pods are in `Running` state.

**Node Reboot Recovery Time**: If the nodes that are part of the air-gapped controller are rebooted, the services and pods will take some time to come back up and become fully operational. This can take approximately **10 to 20 minutes**. Please allow sufficient time for the system to recover before attempting to access the console UI.

### Node Reboot

Always **cordon and drain** nodes before rebooting them.

If a node running pods (especially stateful pods such as PostgreSQL) is rebooted without first being cordoned and drained, pods may fail to restart cleanly. Common symptoms include:

- Pods stuck in `Terminating`
- Stale lock files such as `postmaster.pid` preventing startup

To reboot a node gracefully:

```
   kubectl cordon <node-name>
   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
   # reboot the node
   kubectl uncordon <node-name>
   ```

This evicts pods in an orderly fashion and lets volumes detach properly, so pods reschedule and start cleanly afterward.

---

## Cluster Issues

### Cluster Dashboard Not Loading

**Problem**: Cluster dashboards may not load due to TimescaleDB instability affecting metrics flow.

**Troubleshooting Steps**:

1. Check if required pods are running:

```
   kubectl get pods -A | grep timescale
   kubectl get pods -A | grep promscale
   ```

2. If pods are unhealthy:

3. Review pod logs:

```
   kubectl logs -n <namespace> <timescale-pod-name>
   ```

4. Restart TimescaleDB pods:

```
   kubectl delete pod <timescale-pod-name> -n <namespace>
   ```

**Tip**: Look for connection timeouts or OOM (out of memory) errors in the logs.

---

### MKS Upgrade Failures During Preflight Checks

**Problem**: MKS upgrades may fail intermittently with control channel status fluctuations:

```
Status of control channel for <Node_name>: Down
Status of control channel for <Node_name>: Up
Status of control channel for <Node_name>: Up
Status of control channel for <Node_name>: Up
```

**Solution**:

1. SSH into the affected MKS node.
2. Restart the required services:

```
   sudo systemctl restart salt-minion
   sudo systemctl restart chisel.service
   ```

**Note**: Wait a few seconds after restarting services before re-attempting the upgrade.

---

### ZTKA Connection Issues

**Problem**: Unable to access cluster using ZTKA channel (kubectl hangs or times out).

**Troubleshooting Steps**:

1. Verify all cluster pods are running.
2. Check relay-agent pod logs:

```
   kubectl logs -n rafay-system -l app=relay-agent
   kubectl logs -n rafay-system -l app=v2-relay-agent
   ```

**Common Causes**:
- Firewall rules blocking outbound controller connections
- DNS or networking issues in restricted environments

**Solution**:

Restart the relay-agent pod:

```
   kubectl delete pod -n rafay-system -l app=relay-agent
   ```

**Tip**: After restart, verify kubectl connectivity through ZTKA channel.