Every new version of Kubernetes adds more features and functions. However most of those are released in a controlled fashion, where while these features are in alpha or beta, they must be manually enabled using so called Feature Gates. This post will show how to enable feature gates on a live Kubernetes cluster.

About Feature Gates

You can find more on Feature Gates here:
https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

For this blog I want to enable the Storage Feature Gates for Snapshots and Clones (VolumeSnapshotDataSource and VolumePVCDataSource). If you don’t have a live/running Kubernetes cluster, you could just specify the Feature Gates during the deployment as I showed here. However there is no reason to redeploy a cluster, just to enable (or disabled) these Feature Gates.

Enable Feature Gates for kubelet

The first step is to enable the Feature Gates for the kubelet service on all nodes in the cluster.

sudo vi /etc/kubernetes/kubelet.env

In this file you’ll find the startup parameters for kubelet. Add the line --feature-gates=VolumeSnapshotDataSource=True,VolumePVCDataSource=True \ to the KUBELET_ARGS portion as shown below:

KUBELET_ARGS="--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \
--config=/etc/kubernetes/kubelet-config.yaml \
--kubeconfig=/etc/kubernetes/kubelet.conf \
--pod-infra-container-image=k8s.gcr.io/pause:3.1 \
--runtime-cgroups=/systemd/system.slice \
--feature-gates=VolumeSnapshotDataSource=True,VolumePVCDataSource=True \
   "

To activate these changes, we need to restart the kubelet process. Restarting the kubelet process does not affect running pods, since kubelet is only the management interface for the Kubernetes node. The action pods are using the container runtime engine, which we will not touch here.

sudo systemctl restart kubelet

To check if the change was successful, execute the following command:

ps aux | grep kubelet | grep feature-gates

dnix@node1:~$ ps aux | grep kubelet | grep feature-gates
root     10556  8.0 21.1 635156 431272 ?       Ssl  08:50   0:59 kube-apiserver --advertise-address=10.1.1.111
...
--runtime-cgroups=/systemd/system.slice --feature-gates=VolumeSnapshotDataSource=True,VolumePVCDataSource=True --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin

I’ve removed some lines in the output above, for readability. However you should see the --feature-gates line activated for the kubelet process.

Make sure you repeat the steps above for all Kubernetes nodes in the cluster.

Enable Feature Gates for API server

Now that our Kubernetes nodes access the Feature Gates, we also want to expose the Feature Gates via the Kubernetes API. For this we also need to add the Feature Gates to the API server.

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add the line --feature-gates=VolumeSnapshotDataSource=True,VolumePVCDataSource=True to the kube-apiserver section as shown below. I’ve removed some lines for readability.

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.1.1.111
...
    - --feature-gates=VolumeSnapshotDataSource=True,VolumePVCDataSource=True

Once you save your changes, the API server will automatically restart to activate these changes. To make sure everything went well, use the following to check if the Feature Gates are enabled:

ps aux | grep apiserver | grep feature-gates

This should result in something like this (I again removed some of the output for readability:

rdeenik@node2:~$ ps aux | grep apiserver | grep feature-gates
root     10556  7.7 21.1 635156 432264 ?       Ssl  08:50   1:03 kube-apiserver --advertise-address=10.1.1.111
...
--tls-private-key-file=/etc/kubernetes/ssl/apiserver.key --feature-gates=VolumeSnapshotDataSource=True,VolumePVCDataSource=True

Conclusion

And with that, we’ve enable our Feature Gates on a live Kubernetes cluster. We are now ready to start working with the new features, in this cas Snapshots and Clones.