Remove orphaned CNS Container Volumes from vSphere 6.7U3

In the moment I’m learning Kubernetes from scratch and when I tried to implement vSphere CSI provider, I run in some issues which results into orphaned persistent volumes. Even after fixing the problem and cleaning up everything in my Kubernetes cluster, I had several CNS volumes left.

If you compare this screenshot with the command output, you’re able to recognize that in the screenshot there are some more.

marco@lab-kube-m1:~$ kubectl get pv -A
NAME                                       CAPACITY ACCESS MODES RECLAIM POLICY   STATUS   CLAIM ...
pvc-17d53505-a5ec-42fa-a6f0-647cfe8203d6   1Gi      RWO          Delete           Bound    mongodb/mongodb-persistent-storage-claim-mongod-1 ...
pvc-bef3cda1-4d9e-483c-b5ae-1cb461646614   1Gi      RWO          Delete           Bound    mongodb/mongodb-persistent-storage-claim-mongod-2 ...
pvc-d560a6ff-d10c-4098-875f-4921d988fe92   1Gi      RWO          Delete           Bound    mongodb/mongodb-persistent-storage-claim-mongod-0

In my homelab I actually run vSphere 6.7U3. I needed some time to figure out how I can remove this CNS volumes, but it seems that VMware only had added an option to remove this orphaned CNS volumes in vSphere 7.0U2. Because the other software in my homelab is not allowing to upgrade my version, I needed to search for another solution. And I found a hint.

First of all we need to install govc on our system. I used my Windows Server 2019 in Homelab for this and downloaded the most current release of the precompiled binary. You find it here: https://github.com/vmware/govmomi/releases

I downloaded this binary to “C:\bin\” and started an command line. In this command I need to set several temporary variables, which are used for vCenter authentication.

C:\bin> set GOVC_URL=lab-vc01.homelab.horstmann.in
C:\bin> set GOVC_USERNAME=Administrator@vsphere.local
C:\bin> set GOVC_PASSWORD=XXXXXXXXX
C:\bin> set GOVC_INSECURE=true

After setting this variables, we can run the govc binary with several parameters to reconcile the datastore content. In my case I had to set the datacenter name (Homelab), which I use in my vCenter, and name of the datastore, where this orphaned CNS volumes has been shown.

C:\bin>govc.exe volume.ls
2010194c-263d-4a22-9ffc-82947ead57c7  pvc-324ee877-148e-4f3c-9557-95554e251cd5
698e117e-e640-4581-a9f3-5e499b6f1101  pvc-bef3cda1-4d9e-483c-b5ae-1cb461646614
cc0f96e2-f850-491d-8786-17a30120e6d3  pvc-17d53505-a5ec-42fa-a6f0-647cfe8203d6
f7649917-57dd-43fd-9b70-bd968b484600  pvc-d560a6ff-d10c-4098-875f-4921d988fe92

Lets say we want to delete pvc-d560…. then we need to lookup the volume id from the output before and run this command.

C:\bin>govc.exe volume.rm f7649917-57dd-43fd-9b70-bd968b484600

Ok, that’s all and easier than I expected, but it was hard to figure out how this can be cleaned up. I hope this helps you to clean up your orphaned CNS volumes.