How I built my Kubernetes Homelab – Part 4

After my Kubernetes cluster is up and running I want to do the next step. As a former storage guy I need storage. The running applications as well. Without the option to save and snapshot data to persistent storage all I do inside of a pod is gone when it will be redeployed and this maybe happens quite often.

As I mentioned before I have here a little Homelab running on vSphere 6.7 Update 3, which can act as Cloud Native Storage (CNS). CNS is a features which makes Kubernetes aware of how it can provision storage on-demand on vSphere. This in a fully automated and scalable fashion.

Installing the vSphere Cloud Provider Interface (CPI)

First of all we need to create a secret, where we store user credentials for vCenter access. In my lab I used my existing Administrator account. Maybe you want to build it more secure, then you can use this VMware Documentation to create a role for a user with only necessary rights.

marco@lab-kube-m1:~$ cat cpi-global-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cpi-global-secret
  namespace: kube-system
stringData:
  lab-vc01.homelab.horstmann.in.username: "administrator@vsphere.local"
  lab-vc01.homelab.horstmann.in.password: "XXXXXXXX"

marco@lab-kube-m1:~/yaml/vcenter$ kubectl create -f cpi-global-secret.yaml
secret/cpi-engineering-secret created

We can verify that this credentials secret is successfully created in the kube-system namespace with this command.

marco@lab-kube-m1:~$ kubectl get secret cpi-global-secret --namespace=kube-system
NAME                TYPE     DATA   AGE
cpi-global-secret   Opaque   2      3m

The next step is that we create a configmap for vSphere. These vsphere.conf we create in this step can stored anywhere (exept /etc/kubernetes/) . This configmap will provide informations to the CPI components to access the vCenter.

marco@lab-kube-m1:~/# cat vsphere.conf
# Global properties in this section will be used for all specified vCenters unless overriden in VirtualCenter section.
global:
  port: 443
  # set insecureFlag to true if the vCenter uses a self-signed cert
  insecureFlag: true
  # settings for using k8s secret
  secretName: cpi-global-secret
  secretNamespace: kube-system

# vcenter section
vcenter:
  tenant-homelab:
    server: lab-vc01.homelab.horstmann.in
    datacenters:
      - Homelab


marco@lab-kube-m1:~$ kubectl create configmap cloud-config --from-file=vsphere.conf --namespace=kube-system

Before we installing the vSphere Cloud Controller Manager we need to make sure that all nodes are tainted with “node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule”. When configuring the cluster with an external cloud provider this marks the nodes as unusable. The cloud provider how initialzes this node will remove the taint to make the worker node usable.

marco@lab-kube-m1:~$ kubectl taint node lab-kube-n1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
marco@lab-kube-m1:~$ kubectl taint node lab-kube-n2 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# We can check if the the taint is set. 
marco@lab-kube-m1:~$ kubectl describe nodes | egrep "Taints:|Name:"
Name:               lab-kube-m1
Taints:             node-role.kubernetes.io/master:NoSchedule
Name:               lab-kube-n1
Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
Name:               lab-kube-n2
Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

To install the vSphere Cloud Provider, we have to deploy 3 manifests. This three manifests will create the RBAC roles and bindings as well as it deploys the Cloud Controller Manager in a DaemonSet.

marco@lab-kube-m1:~$  kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/master/manifests/controller-manager/cloud-controller-manager-roles.yaml
clusterrole.rbac.authorization.k8s.io/system:cloud-controller-manager created

marco@lab-kube-m1:~$  kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml
clusterrolebinding.rbac.authorization.k8s.io/system:cloud-controller-manager created

marco@lab-kube-m1:~$  kubectl apply -f https://github.com/kubernetes/cloud-provider-vsphere/raw/master/manifests/controller-manager/vsphere-cloud-controller-manager-ds.yaml
serviceaccount/cloud-controller-manager created
daemonset.extensions/vsphere-cloud-controller-manager created
service/vsphere-cloud-controller-manager created

To check if the Cloud Provider Manager was sucessfully deployed you can use this command to check if the pod is up and running.

marco@lab-kube-m1:~$ kubectl get pods --namespace=kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
[...]
vsphere-cloud-controller-manager-9xc8n    1/1     Running   9          9d

After the vSphere Cloud Controller Manager is up and running the taints for our nodes should be removed and the output of your cluster nodes should look similar.

marco@lab-kube-m1:~$ kubectl describe nodes | egrep "Taints:|Name:"
Name:               lab-kube-m1
Taints:             node-role.kubernetes.io/master:NoSchedule
Name:               lab-kube-n1
Taints:             <none>
Name:               lab-kube-n2
Taints:             <none>

Every node should now have a ProviderID in it’s specs. If this is missing, like in my case because of mistakes in pod networking configuration, you can set the UUIDs manually. Without this information the CSI provider is not working correctly and e.g. cannot attach persistent volumes to our worker nodes. I plan to create for this a separate blog post.

marco@lab-kube-m1:~$ kubectl describe nodes | egrep "Name:|ProviderID:"
Name:               lab-kube-m1
ProviderID:                   vsphere://421a73f1-eb70-ae31-8ce0-78a38a0387f2
Name:               lab-kube-n1
ProviderID:                   vsphere://421afd60-e13c-fcc5-b6fb-2f2e6b749c1a
Name:               lab-kube-n2
ProviderID:                   vsphere://421aadff-58d3-eeed-89e8-6df01b1c1713

Install vSphere Container Storage Interface Driver

We need to have an additional configuration file to provide access credentials and defintion of the vCenter details. I created a file “/etc/kubernetes/csi-vsphere.conf” and used it to generate a secret in the kube-system namespace.

marco@lab-kube-m1:~/# cat /etc/kubernetes/csi-vsphere.conf
[Global]
cluster-id = "Homelab"
cluster-distribution = "Ubuntu"
#ca-file = <ca file path> # optional, use with insecure-flag set to false
#thumbprint = "<cert thumbprint>" # optional, use with insecure-flag set to false without providing ca-file

[VirtualCenter "lab-vc01.homelab.horstmann.in"]
insecure-flag = true
user = "Administrator@vsphere.local"
password = "XXXXXXXX"
port = "443"
datacenters = "Homelab"
marco@lab-kube-m1:~$ kubectl create secret generic vsphere-config-secret --from-file=/etc/kubernetes/csi-vsphere.conf --namespace=kube-system

This credentials will be used to deploy the last CSI components. It deploys the RBAC roles for the vSphere CSI Controller, creates an deployment for this vSphere CSI controller and deploys a Daemonset for CSI node, which will run on every node.

marco@lab-kube-m1:~$ https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/master/manifests/v2.1.1/vsphere-67u3/vanilla/rbac/vsphere-csi-controller-rbac.yaml

marco@lab-kube-m1:~$ kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/master/manifests/v2.1.1/vsphere-67u3/vanilla/deploy/vsphere-csi-controller-deployment.yaml

#There is also a CSI node Daemonset to be deployed, that will run on every node.
marco@lab-kube-m1:~$ kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/master/manifests/v2.1.1/vsphere-67u3/vanilla/deploy/vsphere-csi-node-ds.yaml

We can check with this command if all pods are up and running.

marco@lab-kube-m1:~$ kubectl get pods --namespace=kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
[...]
vsphere-csi-controller-66c54895c5-rtjfb   5/5     Running   17         6m19h
vsphere-csi-node-54fmh                    3/3     Running   0          8m
vsphere-csi-node-d2s4l                    3/3     Running   0          8m
vsphere-csi-node-ddrtq                    3/3     Running   4          8m

Now everything is created to connect and manage persistent volumes on vSphere. The only missing part is now to create a default storage class for deploying persistent volumes.

Create a “default” Storage Class

When we want to deploy persistent volumes from vSphere we need to create a storage class. But why do we need a storage class? A Storage Class allows us to deploy applications to a space with specific parameters. A classic example could be to have two storage classes called “ssd” and “hdd” and allow application owners to deploy their application to a storage class with high or low IO capacity.

To create an own storage class we need to create a VMware Tag and a VM Storage Policy. We create here a Tag category called “Storage-Class”. This Tag category can only be used on Datastores in my case and only one tag can be applied to each object.

After creating the Tag Category we create one or more Tags in this catagory. In my lab we use an NFS Export of a old QNAP system, which is mounted as a VMware Datastore. Because of this I called my Tag “NAS”.

After creating the Tags, we can create a VM Storage Policy. The name of this VM Storage Policy “NAS Storage” will be later used in a manifest to create the Kubernetes Storage class.

I use in my lab a QNAP NFS export as datastore. Because of this we need to select “Enable tag based placement rules”. If you are using a vSAN-based storage you need to select the other option. You can find here, how to use vSAN to provision storage.

We need here to select the Tag category and “Use storage tagged with”. With Brose Tags we can select the tag “NAS”.

The wizard will now show us if our datastores are compatible with this configuration.

Just a short recap of what we have configured in this VM Storage Policy.

Ok, on VMware’s side everything is configured. How we get this mapped to kubernetes? Again we have to create a manifest file called e.g. nas.yaml. In this manifest we select a name, make this storage class the default, set the storagepolicyname according our vSphere VM Storage Policy and set the filesystem type. Now we can deploy this manifest with kubectl and check if it was created.

#Content of nas.yaml file
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: nas-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "NAS Storage"
  fstype: ext4
marco@lab-kube-m1:~$ kubectl create -f nas.yaml
storageclass.storage.k8s.io/nas-sc created

marco@lab-kube-m1:~$ kubectl get storageclass
NAME               PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nas-sc (default)   csi.vsphere.vmware.com   Delete          Immediate           false                  2s

Everything is now ready to deploy our first own application to our Kubernetes cluster. Please join me in my next part of this series where we will create a demo application based on mysql and phpMyAdmin I want to use for demonstration purposes.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

nineteen − 10 =