# Kubernetes Component Configuration

## Kubelet Arguments

**Kubelet settings** can be customized to optimize configurations based on workload and operational requirements. Kubelet parameters are exposed in both cluster and node settings, allowing configuration at a global or node-specific level. In addition to `key-value` Kubelet arguments, structured KubeletConfiguration YAML is supported to configure any valid upstream Kubelet parameter.

Cluster-level settings are applied to all nodes. When a node defines its own configuration, the node-level settings take precedence for matching fields. For fields that do not exist at the node level, the cluster-level configuration continues to apply. Each node receives a merged configuration using this priority model.

**_Types of Kubelet Configuration_**

Kubelet configuration is supported through two mechanisms, depending on the level of control required:

### 1. **Kubelet Extra Args (Key–Value Arguments)**

Kubelet Extra Args allow configuring command-line style Kubelet parameters using simple key-value pairs. These fields are suitable for flat arguments such as `--max-pods`, timeouts, feature gates, and other non-nested flags. Multiple arguments can be added in the **Add/Edit Kubelet Extra Args** dialog, and the settings are applied at the cluster or node level.

This method is intended for straightforward tuning where parameters can be expressed as single key-value entries.

### 2. **KubeletConfiguration Overrides (YAML-Based)**

KubeletConfiguration Overrides provide support for the full upstream **KubeletConfiguration** API and allow defining structured YAML, including nested fields, boolean policies, durations, and advanced configuration blocks. This method enables configuring any valid Kubelet parameter, including complex fields that cannot be expressed through simple key-value arguments.

Overrides can be applied at both the cluster and node level, with node-level settings taking precedence when both levels define the same fields.

**YAML Reference Example**

A built-in reference panel is available to help construct valid KubeletConfiguration YAML. Selecting the **help** icon in the editor opens a read-only example based on the upstream `KubeletConfiguration` API. This example includes commonly used fields such as authentication, authorization, cgroup settings, cluster DNS configuration, and other standard parameters defined by Kubernetes.

The reference panel is intended as a guide for understanding the structure and formatting of the KubeletConfiguration specification. Values can then be added or modified in the editable YAML editor on the right, where `cluster-level` or `node-level` overrides are entered.

This reference is provided to ensure that `KubeletConfiguration Overrides` follow the correct structure, field types, and formatting as defined by Kubernetes.

The following are the default Kubelet arguments configured by Rafay. These arguments cannot be overridden, and attempting to override them will result in deployment failure. However, additional arguments can be added if needed.

- `feature-gates=ExecProbeTimeout=false`
- `register-with-taints=rafay.dev/Provisioning:NoSchedule`
- `node-ip=10.0.0.229`
- `runtime-request-timeout=15m`
- `container-runtime-endpoint=unix:///run/containerd/containerd.sock`

**_KubeletConfiguration Examples_**

The following examples illustrate commonly used KubeletConfiguration fields that can be applied at either the cluster level or node level.

Example 1: Enable `singleProcessOOMKill` for selected nodes

```
singleProcessOOMKill: true
```

This configuration can be added as a node-level override so that the setting applies only to the selected nodes.

Example 2: Configure `imageMaximumGCAge`

```
imageMaximumGCAge: "1h"
```

Example 3: Combined advanced parameters

```
maxPods: 500
imageMinimumGCAge: "2m2s"
imageGCHighThresholdPercent: 80
```

Click **Save** after updating the configuration.

Kubelet Arguments can be added, edited, or deleted on **Day 2**. Kubelet Arguments cannot be modified at both the cluster level and node level simultaneously during Day 2 updates; changes can be applied either at the cluster level or in bulk for selected nodes. KubeletConfiguration Overrides are also supported on Day 2 and can be updated at the cluster level or node level as required.

## Control Plane Overrides

Control Plane Overrides enable advanced customization of Kubernetes control plane components in upstream clusters. This feature allows modification of control plane arguments, volumes, and volume mounts without exposing or editing the full static pod manifest files.

This capability is supported **only for control plane nodes** and does not apply to worker nodes. There is no node-level configuration for this feature. All overrides are applied at the cluster level.

**Purpose**

Control Plane Overrides provide a controlled mechanism to customize Kubernetes control plane behavior while preserving required defaults and overall cluster stability.

This feature can be used to:

- Enable or disable feature gates
- Modify admission plugins
- Adjust logging verbosity
- Configure audit logging parameters
- Add or override volume definitions
- Update volume mount paths
- Apply organization-specific configuration requirements

The complete YAML manifest for control plane components is not exposed. Only specific override sections are configurable.

**What Can Be Overridden**

Only the following sections of the control plane manifests can be modified:

1. Extra args (key-value pairs)
2. Extra volumes
3. Extra volume mounts

Other sections such as container image, probes, resource settings, security context, networking configuration, and priority settings are not configurable.

Only the fields that require customization need to be provided. Existing values remain unchanged unless explicitly overridden.

**Scope and Application**

- Control Plane Overrides are cluster-level settings.
- Overrides apply uniformly to all control plane nodes.
- If multiple control plane nodes exist, the same configuration is applied to each API server, controller manager, and scheduler instance.
- Supported during cluster creation (Day 0).
- Supported during post-creation updates (Day 2).

### Editing Control Plane Overrides

Selecting the **Edit** option under Control Plane Overrides opens a tabbed configuration panel.

Separate tabs are available for:

- Kube API Server
- Kube Controller Manager
- Kube Scheduler

Each tab provides:

- Extra args
- Extra volumes
- Extra volume mounts

Configuration entered in one tab applies only to that component.

### Extra Args

Extra args allow configuration of control plane arguments using key-value pairs.

Each entry consists of:

- **Key** – Argument name
- **Value** – Argument value

**Behavior**

- If the key already exists, its value is overridden.
- If the key does not exist, it is added.
- Existing values are replaced, not merged.
- Only the fields that need to be customized must be provided.

**_Example 1: Simple Argument_**

UI entry:

| Key      | Value  |
|----------|--------|
| profiling | false  |

**Result**

The existing `profiling` setting is updated to `false` for the selected component across all control plane nodes.

**_Example 2: Comma-Separated Argument_**

Certain arguments accept comma-separated values, such as:

- `enable-admission-plugins`
- `feature-gates`
- `tls-cipher-suites`

When modifying such arguments:

- The complete existing value must be copied.
- Any new value must be appended using a comma.
- Providing only a new value replaces the entire list.

Example UI entry:

| Key                          | Value  |
|------------------------------|--------|
| enable-admission-plugins     | NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota,NodeRestriction,EventRateLimit,PodSecurity |

**Result**

The existing list is fully replaced with the provided value, preserving required entries and appending the new plugin.

If only `PodSecurity` is provided, all previously configured plugins are removed and cluster functionality may break.

This rule applies to all comma-separated arguments.

For supported parameters and default values, refer to the [official Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver).

### Extra Volumes

The Extra volumes section accepts YAML input in string format.

Example:

```
- name: safe-dir
  emptyDir: {}
```

**Mandatory Rules**

- Each entry must begin with a hyphen (`-`).
- YAML must be correctly formatted.
- The `name` field is the matching key.

If formatting is incorrect or the hyphen is omitted, the control plane manifest may become corrupted and functionality may be affected.

**Matching Behavior**

- If the `name` matches an existing volume, that volume definition is overridden.
- If the `name` does not exist, a new volume is added.

The `name` field determines whether the configuration overrides or adds a volume.

### Extra Volume Mounts

The Extra volume mounts section also accepts YAML input in string format.

Example:

```
- name: safe-dir
  mountPath: /tmp/safe-apiserver
  readOnly: false
```

**Rules**

- Each entry must begin with a hyphen (`-`).
- The `name` field determines matching behavior.
- Matching name overrides the existing mount configuration.
- A new name adds a new mount entry.

Volumes and volume mounts must use consistent `name` values.

### Cluster Specification Example

Control plane overrides are defined in the cluster specification under `controlPlaneOverrides`

Below is an example spec:

```
apiVersion: infra.k8smgmt.io/v3
kind: Cluster
metadata:
  name: test-pfug-2
  project: defaultproject
spec:
  blueprint:
    name: minimal
  config:
    autoApproveNodes: true
    kubernetesVersion: v1.33.5
    installerTtl: 365
    platformVersion: v0.1.0
    kubeletExtraArgs:
      max-pods: "600"
    controlPlaneOverrides:
      kubeApiserver:
        extraArgs:
          profiling: "false"
          v: "4"
        extraVolumeMounts: |
          - name: safe-dir
            mountPath: /tmp/safe-apiserver
            readOnly: false
        extraVolumes: |
          - name: safe-dir
            emptyDir: {}
      kubeControllerManager:
        extraArgs:
          profiling: "false"
          v: "2"
        extraVolumeMounts: |
          - name: safe-dir
            mountPath: /tmp/safe-controller
            readOnly: false
        extraVolumes: |
          - name: safe-dir
            emptyDir: {}
      kubeScheduler:
        extraArgs:
          profiling: "false"
          v: "2"
        extraVolumeMounts: |
          - name: safe-dir
            mountPath: /tmp/safe-scheduler
            readOnly: false
        extraVolumes: |
          - name: safe-dir
            emptyDir: {}
  network:
    cni:
      name: Calico
      version: 3.28.1
    podSubnet: 10.244.0.0/16
    serviceSubnet: 10.96.0.0/12
  nodes:
    - arch: amd64
      hostname: vih-12
      operatingSystem: Ubuntu22.04
      privateip: 10.0.0.128
      roles:
        - Master
        - Worker
        - Storage
      ssh:
        ipAddress: 129.146.39.135
        port: "22"
        privateKeyPath: /Users/vihari/.ssh/vihari_oci_ssh
        username: ubuntu

type: mks
```

### Day 2 Operations

Control Plane Overrides can be updated after cluster provisioning.

Go to **Clusters → Select Cluster → Configuration → Control Plane Overrides → Edit**.

**Viewing Applied Configuration**

The applied control plane configuration can be viewed from the **Nodes** tab.

Go to **Clusters → Select Cluster → Nodes → Node Actions → View Control Plane Config**.

This opens a read-only view of the Kube API Server, Controller Manager, and Scheduler configuration for that node. Editing is not supported from this screen.

A read-only view of the full control plane YAML is available for each control plane node. This allows verification that overrides have been applied correctly without accessing the node directly.

**Important Considerations**

- Only arguments, volumes, and volume mounts are configurable.
- Comma-separated arguments must include the complete existing value.
- The `name` field determines override versus addition for volumes and mounts.
- Overrides apply uniformly to all control plane nodes.
- Incorrect configuration may impact cluster functionality.