Remove orphaned CNS Container Volumes from vSphere 6.7U3
In the moment I’m learning Kubernetes from scratch and when I tried to implement vSphere CSI provider, I run in some issues which results into orphaned persistent volumes. Even after fixing the problem and cleaning up everything in my Kubernetes cluster, I had several CNS volumes left.
If you compare this screenshot with the command output, you’re able to recognize that in the screenshot there are some more.
marco@lab-kube-m1:~$ kubectl get pv -A NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM ... pvc-17d53505-a5ec-42fa-a6f0-647cfe8203d6 1Gi RWO Delete Bound mongodb/mongodb-persistent-storage-claim-mongod-1 ... pvc-bef3cda1-4d9e-483c-b5ae-1cb461646614 1Gi RWO Delete Bound mongodb/mongodb-persistent-storage-claim-mongod-2 ... pvc-d560a6ff-d10c-4098-875f-4921d988fe92 1Gi RWO Delete Bound mongodb/mongodb-persistent-storage-claim-mongod-0
In my homelab I actually run vSphere 6.7U3. I needed some time to figure out how I can remove this CNS volumes, but it seems that VMware only had added an option to remove this orphaned CNS volumes in vSphere 7.0U2. Because the other software in my homelab is not allowing to upgrade my version, I needed to search for another solution. And I found a hint.
First of all we need to install govc on our system. I used my Windows Server 2019 in Homelab for this and downloaded the most current release of the precompiled binary. You find it here: https://github.com/vmware/govmomi/releases
I downloaded this binary to “C:\bin\” and started an command line. In this command I need to set several temporary variables, which are used for vCenter authentication.
C:\bin> set GOVC_URL=lab-vc01.homelab.horstmann.in C:\bin> set GOVC_USERNAME=Administrator@vsphere.local C:\bin> set GOVC_PASSWORD=XXXXXXXXX C:\bin> set GOVC_INSECURE=true
After setting this variables, we can run the govc binary with several parameters to reconcile the datastore content. In my case I had to set the datacenter name (Homelab), which I use in my vCenter, and name of the datastore, where this orphaned CNS volumes has been shown.
C:\bin>govc.exe volume.ls 2010194c-263d-4a22-9ffc-82947ead57c7 pvc-324ee877-148e-4f3c-9557-95554e251cd5 698e117e-e640-4581-a9f3-5e499b6f1101 pvc-bef3cda1-4d9e-483c-b5ae-1cb461646614 cc0f96e2-f850-491d-8786-17a30120e6d3 pvc-17d53505-a5ec-42fa-a6f0-647cfe8203d6 f7649917-57dd-43fd-9b70-bd968b484600 pvc-d560a6ff-d10c-4098-875f-4921d988fe92
Lets say we want to delete pvc-d560…. then we need to lookup the volume id from the output before and run this command.
C:\bin>govc.exe volume.rm f7649917-57dd-43fd-9b70-bd968b484600
Ok, that’s all and easier than I expected, but it was hard to figure out how this can be cleaned up. I hope this helps you to clean up your orphaned CNS volumes.