# Troubleshooting This section explains the frequently occurred errors during cluster provision --- ## Resource Provisioning Failures ### Scenario 1: Instance Type Not supported The below error is an example that might occur at the time of cluster provision or adding a new nodegroup to the existing cluster **Validation** To overcome this issue, perform the below validations for instance types in a region: - Check your Cloud Credentials (roles based or access id or secret) has the required permission to call ec2 AWS APIs. If the Cloud Credentials are role based, ensure all the appropriate [IAM Policies](https://docs.rafay.co/clusters/eks/iam_policy/) are met - Check whether the configuration has an instance type that is not available in the selected region --- ### Scenario 2: Availability Zones The below error is an example that might occur when the Cloud credentials does not have permission to create resources in the selected region during EKS cluster provision **Validation** Validate the permissions of the cloud credentials used for cluster provisioning to create the resources in that configured region --- ### Scenario 3: Instance Type Permission The below error is an example that might occur when the cloud credentials do not have permission to use a particular instance type, used in the EKS cluster configuration **Validation** - Check for permission and use the right instance type for the cloud credentials - Rectify the permission on AWS to use the required configured instance type --- ### Scenario 4: K8s version upgrade During the k8s version upgrade to 1.25, the below error occurs if the aws-load-balancer-controller version is 2.4.6. The upgrade gets halted and the preflight check fails **Validation** Update the aws-load-balancer-controller to version v2.4.7 and then upgrade the k8s version to 1.25 --- ### Scenario 5: Removal of PSPs The below error is an example that might occur when PSPs are found during the k8s version upgrade to 1.25. **Validation** PSPs are no longer supported in k8s v1.25, hence remove the PSPs and upgrade again --- ## AWS Cloud Errors When provisioning an EKS cluster, it might fail due to various AWS Cloud errors. These errors can stem from resource limitations, network connectivity issues, misconfigurations in the provisioning process, insufficient permissions, service outages impacting required AWS services, software bugs, and region-specific constraints. These factors can disrupt the EKS cluster provisioning process and necessitate troubleshooting to identify and resolve the underlying issues for successful deployment. To gain insight into the failure and its underlying cause, click on **Provision Status** of the failed cluster Click on **Errors** tab and expand the **Cloud Error(s)** section to access detailed information about AWS CloudFormation errors. This action will provide specific details regarding the encountered issues during the cluster provisioning process, enabling you to identify the root cause and take appropriate remedial actions for successful deployment. ### Logs & Events In the event of Cluster Provisioning failure on **Day 0** due to any underlying issues, it's essential to diagnose and resolve them promptly. Along with the **Errors**, users can gain deeper insights into the errors and facilitate debugging by clicking on **Logs & Events** tab to view the CloudFormation stack events from AWS. This action provides access to comprehensive logs dating back to the creation of the stacks, enabling a thorough examination of events leading up to the provisioning failure. This **Logs & Events** are available for the failure scenarios during **Infra Creation and deletion**, **Bootstrap Node creation, and deletion** and **Bootstrap Creation In progress** Logs & Events are available when the cluster provisioning is complete, but the operation status is not ready, even though nodes are being created. This discrepancy might occur if the blueprint status is not initiated when incorrect images are applied. If nodegroup creation fails on **Day 2**, users are allowed to pull the Logs and Events of a specific nodegroup. To access the logs and events pertaining to a specific nodegroup creation failure, click on the corresponding 'nodegroup creation failed' link and review the details. In addition to retrieve logs and events through the user interface, users can also pull cloud logs and events using the [API](https://docs.rafay.co/clusters/eks/api/) and [CLI](https://docs.rafay.co/clusters/eks/declarative/cli/). Important IAM role policy `ec2:GetConsoleOutput` is required to pull bootstrap cloud-init logs. --- ## Provisioning failure with custom KMS key for volume encryption Provisioning an EKS cluster with volume encryption enabled in the node group settings fails. The cluster status shows a bootstrap creation failure with the following error: `Resource handler returned message: "[Issue(Code=AsgInstanceLaunchFailures, Message=Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state, ResourceIds=[eks-bootstrap-]), Issue(Code=NodeCreationFailure, Message=Instances failed to join the kubernetes cluster, ResourceIds=[])]"` ### Validation When a custom KMS key is used for EBS volume encryption, the `AWSServiceRoleForAutoScaling` role must have the required permissions on the KMS key. Without these permissions, instances fail to launch and join the cluster. ### Resolution Ensure that the KMS key policy includes the following statements to allow encryption, decryption, and grant creation for the `AWSServiceRoleForAutoScaling` role. ``` { "Sid": "Allow use of the key", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ], "Resource": "*" }, { "Sid": "Allow attachment of persistent resources", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling" }, "Action": [ "kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant" ], "Resource": "*", "Condition": { "Bool": { "kms:GrantIsForAWSResource": "true" } } } ``` ---