KOP Upstream Clusters - Provisioning - Rafay Product Documentation

Provisioning

The following sequence diagram describes the high-level steps carried out in a sequence during the provisioning process. Customers can optionally automate the entire sequence using the RCTL CLI or REST APIs or their automation tools.

Datacenter (Edge) Admin Rafay Datacenter (Edge) Admin Rafay Upstream Kubernetes Cluster Provisioning Create Cluster Download installer & installer Upload installer & credentials Run Installer Register with Rafay Installer Downloads Software Packages from Rafay Container Registry Installer Deploys k8s software packages Kubernetes Cluster Checks In (Managed by Rafay)

Detailed steps carried out in a sequence during the provisioning process

Rafay Controller Node User Rafay Controller Node User Conjurer Hard Prechecks If auto-approve configured If not auto-approved User configured roles determine actions Single node provisioning steps Node instructions sent by the Controller. Download Conjurer binary from Rafay Controller Run conjurer binary on the node Outbound connection on 443 Port As part of Conjurer run, Prechecks will be done Check for Unsupported Operating System Verify DNS Port Availability Test Connectivity to Controller Ensure Communication with NTP Server Check for Time Skew Verify Previous Kubernetes Installation Validate Hostname Firewalls check(fapolicyd, firewalld) only for linux Check Nameserver Rotation Option Node discovered Automatically approve node Meta information retrieval Manually approve node Node approved Execute role-specific instructions Pull and install binaries (Kubernetes and other required binaries for the node) Install and bring up Consul and etcd Execute kubeadm init Install CNI Untaint the node Install certificate generation component Apply Rafay operator Wait for all pods to be up Mark cluster as READY

Demo Video

Watch a video of provisioning of a "Multi Master" upstream Kubernetes cluster on "CentOS" with only storage for persistent volumes.

STEP 1: Select Cluster Configuration

Review the supported cluster configurations and select your desired cluster configuration. This will determine the number of nodes you need to prepare to initiate cluster provisioning.

Type Number of Initial Nodes
Converged, Single Master 1 Nodes (1 Master/Worker)
Dedicated, Single Master 2 Nodes (1 Master + 1 Worker)
Converged, Multi Master 3 Nodes (3 Masters + 1 Worker)
Dedicated, Multi Master 4 Nodes (3 Masters + 1 Worker)

STEP 2: Prepare Nodes

Create VMs or bare metal instances compatible with the infrastructure requirements. Ensure that you have SSH access to all the instances/VMs

Important
Ensure you have the exact number of nodes for initial provisioning as per the cluster configuration from the previous step. Additional worker nodes can be added once the cluster is successfully provisioned.

Preparing RHEL 8.7 Nodes

For environments using Red Hat Enterprise Linux (RHEL) 8.7, ensure the required packages are pre-installed to avoid dependency issues during cluster provisioning. By staging the base OS with all required dependencies, the provisioning process can skip YUM-based package downloads, improving speed and reliability.

sudo yum update -y && sudo yum install -y \
bzip2 psmisc unzip file lvm2 openssl iscsi-initiator-utils device-mapper-multipath cryptsetup \
curl wget libcgroup checkpolicy container-selinux socat chrony ebtables conntrack \
ca-certificates iptables at bzip2 tc iproute-tc pciutils python36 --allowerasing --nobest && \
systemctl enable chronyd && systemctl start chronyd

This installs all required dependencies and starts the chronyd service for time synchronization. The flags --allowerasing and --nobest help resolve potential package version conflicts on RHEL 8.7.

conjurer install --force

💡 Info
The --force flag helps bypass subscription-related errors if the instance is not attached to a valid Red Hat subscription. Ensure the packages are manually staged as shown above before using this option.

STEP 3: Create a Cluster

General

All the defaults are automatically selected and presented. Several options are available for customization.

Location

Blueprint

Note: On Day 1 operation, you can select any blueprint other than default-upstream if storage node is not available

Kubernetes Version

Platform Version
Defines the version of core components CRI, Orchestration Proxy, ETCD, Cluster utils, and Orchestration Agent that are installed and managed with the cluster.

Refer to the Platform Version documentation for more details. For the platform version support matrix and component versions, see Supported Environments.

Operating System

Cluster Settings

Cluster Settings allow configuration of Cluster Labels, Cluster Annotations, Kubelet Configuration, and Control Plane Overrides during cluster provisioning.

Note:
Annotations can be configured at both the cluster level and node level during cluster creation and through Day 2 operations. Configuration is supported via the UI, Terraform, RCTL, and APIs. Cluster-level annotations are applied to all nodes. When node-level annotations are also defined, the annotations are merged. If the same key exists at both levels, the node-level annotation value takes precedence.

For detailed information about Kubelet configuration and Control Plane Overrides, including key-value arguments, structured YAML overrides, and Day 2 updates, refer to Kubernetes Component Configuration.

Advanced

Integrated Storage

  1. Users can set the storage provider details along with the cluster creation process only when selecting a blueprint other than default-upstream. This is because no storage providers are integrated to those blueprint options.

  2. The Managed Storage is available as an add-on with the blueprint default-upstream, thus the users are not allowed to edit or change the default storage class rook-ceph at the time of cluster creation/provision.

    • To provision a HA Cluster using rook-ceph, it is recommended to have a minimum of three (3) storage nodes
    • To provision a non-HA Cluster using rook-ceph, it is recommended to have a minimum of one (1) storage node

Important
Raw disk partitions are only supported when using the rook-ceph managed add-on for the default-upstream blueprint versions from 1.12 to 1.15. Disk encryption is disabled in these versions to support raw disk partitions.

Security
By default, as a security precaution, nodes need to be approved before joining a cluster. Auto Approval of nodes is available, and this can help streamline the cluster provisioning and expansion workflows.

Kubernetes Masters

Installer Certificate TTL
Optionally, users can set the validity period for the Installer Certificates by specifying the number of days. By default, the expiry is set to 365 days.

For more information, refer to the TTL Certificate page.

HTTP Proxy

Important
Proxy configuration cannot be changed once the cluster is created

Cluster Networking

If you want to customize the subnets used for Pod Networking and K8s Services:

For Cilium:

Note: To provision a windows worker node, it is mandatory to already have a Linux master node (Control Plane).

Refer Add Worker Nodes for more information on adding a window worker node to the upstream cluster

Important
Cluster Networking cannot be changed once the cluster is created

STEP 4: Installer and Credentials

Copy and run the commands that includes the credentials and passphrase.

Note
The option you choose is the one you will use in the subsequent steps.

Linux Architecture

Prepare Nodes

Preflight Checks
It is strongly recommended to perform automated preflight tests on every node to ensure that it has "compatible" hardware, software, and configuration. View the detailed list of preflight checks.

Run Installer

Important
Firewall Configuration for Inter Node Communication:
Ensure that the host firewall is configured to allow inter node communication in multi node setup so that pod's can reach to the api server.

Windows Architecture

Prepare Nodes
Select Windows for the system using windows processor developed by Intel/AMD

To download Conjurer, the credentials, and the passphrase, you can either:

Downloaded FilesRun Command

An illustrative example is provided below. This assumes that you have the three downloaded files in the current working directory. The three files will be securely uploaded to the “/tmp” folder on the instance.

$ scp -i <keypairfile.pem> * ubuntu@<Node's External IP Address>:/tmp

Preflight Checks
It is strongly recommended to perform automated preflight tests on every node to ensure that it has "compatible" hardware, software, and configuration. View the detailed list of preflight checks.

Copy and run the command in terminal

Run Installer

An illustrative example is provided below:

sudo ./conjurer -edge-name="onpremcluster" -passphrase-file="onpremcluster-passphrase.txt" -creds-file="onpremcluster.pem -t

[+] Initiating edge node install

[+] Provisioning node
      [+] Step 1. Installing node-agent
      [+] Step 2. Setting hostname to node-72djl2g-192-168-0-20-onpremcluster
      [+] Step 3. Installing credentials on node
      [+] Step 4. Configuring node-agent
      [+] Step 5. Starting node-agent

[+] Successfully provisioned node

ℹ️ What is 'Conjurer'?
Conjurer is a cluster installer and bootstrap tool developed and maintained by Rafay. It is a proprietary tool, not open source. Conjurer connects and registers nodes with the Rafay Controller by internally bringing up the Salt Minion and Chisel, enabling the connectivity required to provision Kubernetes nodes. It validates node readiness by checking system prerequisites such as storage, DNS, and file system configuration, using activation secrets to access the necessary Controller information and authentication credentials for registration. Once the node passes validation, the node gets registered in the Rafay Controller. This connection allows the node to be used in provisioning Kubernetes clusters and cluster lifecycle management.

Important

Step 5: Salt Minion coexists with customer's salt minion

Conjurer binary has introduced a multi-minion option that allows installing Rafay salt-minion, which can run along with other salt-minion(s), already installed on the node by the customer.

ℹ️ About Salt/Minion
Salt is an open-source automation and infrastructure management tool used to provision Kubernetes clusters on bare metal or virtual machines (VMs). It is released under the Apache 2.0 license and developed by VMware. Salt operates on a master-minion architecture, where the minion runs on the customer's node, and the master runs on the controller. The master orchestrates workflows and sends commands, which are executed by the minion on the node. Salt is leveraged for automating and managing tasks across multiple nodes, including provisioning Kubernetes clusters and handling other infrastructure management tasks. Conjurer brings up the Salt minion on the node, enabling communication between the node and the Rafay Controller. Once the Salt minion is active, the node can receive orchestrated workflows and Kubernetes provisioning tasks from the controller, leveraging Salt’s automation capabilities to manage the node and its tasks.

sudo ./conjurer -m -edge-name="onpremcluster" -passphrase-file="onpremcluster-passphrase.txt" -creds-file="onpremcluster.pem -t

sudo ./conjurer -m -d -edge-name="onpremcluster" -passphrase-file="onpremcluster-passphrase.txt" -creds-file="onpremcluster.pem -t

Once the run conjurer step is complete, the node will show up on the Web Console with the status as DISCOVERED.

STEP 6: Approve Node

This is an optional approval step that acts as a security control to ensure that administrators can inspect and approve a node before it can become part of the cluster.

STEP 7: Configure Node

This is a mandatory configuration step that allows the infrastructure administrator to specify the “role” for the node.

Important
Without the configuration step, cluster provisioning cannot be initiated.

STEP 8: Provisioning

All the necessary configurations are provided, and the Controller can start provisioning Kubernetes with all required software add-ons. These will be automatically provisioned and configured to operationalize the cluster.

Important

Below is an example of an upstream cluster with windows worker node

The Search Nodes box allows users to quickly filter the list of nodes by entering keywords or names, while the Sort By dropdown allows users to organize nodes based on attributes, such as sorting by Host name in ascending order (A-Z). These features improve efficiency in locating and managing nodes.

Note: In the event of insufficient RAM during MKS cluster provisioning, the system dynamically allocates disk space to compensate for the memory shortage. While this process addresses the memory deficit, it's important to note that automatic swap off is disabled to exclusively utilize the available memory, without engaging disk-based swap.

Provisioning Successful

Once all the steps are complete and if the provision process was successful, you should see details about the cluster on the Web Console.

To view the cost details of this cluster, click on the available cost link. This will navigate you to the Cost Explorer page, where you can access detailed cost information for this specific cluster.

Download the cluster configuration formatted for GitOps System Sync, either from the console, using the RCTL CLI, or programmatically through the REST APIs.

On successful upstream cluster provision, users can view the detailed operations and workflow of the cluster by clicking the Operation Status Ready icon. The screen shows all the stages that occurred during cluster deployment

Users can view the Nodes status and health on this page.

If you encounter any provisioning or upgrade issues, check our Troubleshooting Guide.

Reset Cluster

Post-provisioning, users are allowed to RESET the upstream cluster to reuse the same cluster object in the console while a master node is reinstalled/re-provisioned.

The below message appears to confirm the deletion

On successful cluster reset, nodes are deleted and you will see the below screen with no nodes

Important
The 'Reset Cluster' option is only visible if the user enables the reset flag; otherwise, it does not appear.

Troubleshooting

Once the "minion/node agent" is installed on the node by conjurer successfully and is registered with the controller, it establishes a "long running" web socket with the controller providing "continuous updates" to the controller about progress and status. This information is then presented to authorized administrators via the Console for insights. Optionally, administrators can also view the logs generated by the minion/node agent for detailed visibility.

tail -f /opt/rafay/salt/var/log/salt/minion

View Cluster Configuration

Administrators can view the provisioned cluster's configuration by clicking on the cluster and selecting the "Configuration" tab.