How I built my Kubernetes Homelab – Part 3

Hello again, welcome to part 3 of my little Kubernetes Homelab series. In this part we will setup the Kubernetes cluster itself. Because I want to use volumes created by a VMware vSphere provider and the first steps for this needs to done at the cluster initialization.

First of all we need to create a new file called vsphere.conf in the /etc/kubernetes folder. In this configuration file we are telling Kubernetes which settings should be used to connect to the vCenter for discovering the VMware-specific information.

marco@lab-kube-m1:~# sudo tee /etc/kubernetes/vsphere.conf >/dev/null <<EOF
[Global]
user = "administrator@vsphere.local"
password = "XXXXXXXX"
port = "443"
insecure-flag = "1"

[VirtualCenter "lab-vc01.homelab.horstmann.in"]
datacenters = "Homelab"

[Workspace]
server = "lab-vc01.homelab.horstmann.in"
datacenter = "Homelab"
default-datastore = "QNAP:nfs-vms"
resourcepool-path = "Cluster/Resources"
folder = "k8s"

[Disk]
scsicontrollertype = pvscsi

EOF 

I used this article to create a config file, but needed to modify the config file and migrate the config file to the new format/version because the instructions are outdated. If you need to migrate a config file from one to another kubernetes version, you can use this command to make the conversion to a new Kubernetes version.

kubeadm config migrate --old-config /etc/kubernetes/kubeadminitmaster.yaml --new-config /etc/kubernetes/kubeadminitmasternew.yaml

I used the following configuration file for creating my Kubernetes cluster.

marco@lab-kube-m1:~# sudo tee /etc/kubernetes/kubeadminit.yaml >/dev/null <<EOF
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: y7yaev.9dvwxx6ny4ef8vlq
  ttl: 0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.30.50
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  kubeletExtraArgs:
    cloud-config: /etc/kubernetes/vsphere.conf
    cloud-provider: vsphere
  name: lab-kube-m1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  extraArgs:
    cloud-config: /etc/kubernetes/vsphere.conf
    cloud-provider: vsphere
  extraVolumes:
  - hostPath: /etc/kubernetes/vsphere.conf
    mountPath: /etc/kubernetes/vsphere.conf
    name: cloud
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager:
  extraArgs:
    cloud-config: /etc/kubernetes/vsphere.conf
    cloud-provider: vsphere
  extraVolumes:
  - hostPath: /etc/kubernetes/vsphere.conf
    mountPath: /etc/kubernetes/vsphere.conf
    name: cloud
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.20.6
networking:
  dnsDomain: cluster.local
  podSubnet: 10.10.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}
EOF

Now I will reloaded the services to make sure everything is fine.

marco@allthreenodes:~# sudo systemctl daemon-reload
marco@allthreenodes:~# sudo systemctl restart kubelet

Before I can initialize the first (master) node, kubeadm needs to pull several packages from the Google Container Registry to deploy later the cluster services. This packages must be downloaded on every cluster node.

marco@allthreenodes:~# sudo kubeadm config images pull
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.14.2
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.14.2
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.14.2
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.14.2
[config/images] Pulled k8s.gcr.io/pause:3.1
[config/images] Pulled k8s.gcr.io/etcd:3.3.10
[config/images] Pulled k8s.gcr.io/coredns:1.3.1 

Ok, now this is the very uneventful creation of the cluster. This command runs on the master node and create the whole cluster.

marco@lab-kube-m1:~# sudo kubeadm init --config /etc/kubernetes/kubeadminit.yaml
W0313 00:45:46.625707   18261 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.20.6
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [lab-kube-m1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.10.0.1 192.168.30.50]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [lab-kube-m1 localhost] and IPs [192.168.30.50 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [lab-kube-m1 localhost] and IPs [192.168.30.50 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0313 00:46:19.803820   18261 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0313 00:46:19.804774   18261 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 71.503600 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node lab-kube-m1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node lab-kube-m1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: y7yaev.9dvwxx6ny4ef8vlq
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!
[...]

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.30.50:6443 --token y7yaev.9dvwxx6ny4ef8vlq \
    --discovery-token-ca-cert-hash sha256:e49d5d5b84fadea7affed6382e834bb3cba42a21cb14e544b8832668e317c37e

Because I want to manage the cluster with my normal user I’ve copied the admin.conf in a .kube folder of my home directory. After this I was able to use the kubectl command from my standard user.

marco@lab-kube-m1:~# mkdir -p $HOME/.kube
marco@lab-kube-m1:~# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
marco@lab-kube-m1:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config

Before I join my worker nodes I need to create the pod networking (CNI). I used Calico to provide the connectivity to the pods. In the first step we need to download the calico manifest (YAML) and modify it to match to our pod networking. Otherwise you will stay up until the night is nearly over and don’t know why later steps didn’t work. Especially when the kids get up early in the morning. ;-)

marco@lab-kube-m1:~# wget https://docs.projectcalico.org/manifests/calico.yaml

After downloading the file we need to edit it with e.g. VIM and search for this two lines. By default they are commented out. I changed it
the same value of podSubnet which was configured in my kubeadminit.yaml.

marco@lab-kube-m1:~# vi calico.yaml
[...]
            - name: CALICO_IPV4POOL_CIDR
              value: "10.10.0.0/16"
[...]

Now we need to apply the calico manifest. If you are getting something like syntax failure you probably had only removed the “#” in front of this two lines. But you need to remove “#<space>” to get the format of the YAML syntax right (yaml sucks sometimes)

marco@lab-kube-m1:~# kubectl apply -f calico.yaml

On both worker nodes I ran this command to join the nodes to the Kubernetes cluster with the command the master node told me at the end of the kubeadm init command.

marco@lab-kube-nX:~# sudo kubeadm join 192.168.30.50:6443 --token y7yaev.9dvwxx6ny4ef8vlq --discovery-token-ca-cert-hash sha256:e49d5d5b84fadea7affed6382e834bb3cba42a21cb14e544b8832668e317c37e

You can check if your cluster is running with this command.

marco@lab-kube-m1:~$ kubectl get pods --namespace=kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-69496d8b75-bm2v4   1/1     Running   0          20h
calico-node-jp89s                          1/1     Running   0          20h
calico-node-pkmhl                          1/1     Running   0          20h
calico-node-v7hdf                          1/1     Running   0          20h
coredns-74ff55c5b-nhtw6                    1/1     Running   0          20h
coredns-74ff55c5b-tmmpl                    1/1     Running   0          20h
etcd-lab-kube-m1                           1/1     Running   0          20h
kube-apiserver-lab-kube-m1                 1/1     Running   0          20h
kube-controller-manager-lab-kube-m1        1/1     Running   0          20h
kube-proxy-7xchf                           1/1     Running   0          20h
kube-proxy-9cnk9                           1/1     Running   0          20h
kube-proxy-cpxpn                           1/1     Running   0          20h
kube-scheduler-lab-kube-m1                 1/1     Running   0          20h

With this commands I got my cluster up and running. I hope my journey blog is useful for you. In the next part of this series I will deploy all necessary components to discover storage relevant components to use the vSphere Cloud Native Storage (CNS) with the vSphere CSI provider.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

2 × three =