Volumes in Kubernetes

Volume management in Kubernetes requires considering what type of volume you want to set up and how the volume will be created/provided. I will focus mostly on persistent volumes, but also touch on at least hostpath volumes.

What are Volumes

Volumes are storage that persists as long as a pod exists, and is available to all containers in that pod. It is deleted when the pod is deleted. these are ephemeral volumes that usually will use local or host storage since they are not indicated to last beyond the lifetime of a pod. hostpath, local, emptydir are common types for this.

PersistentVolumes

A PersistentVolume (PV) is a piece of storage like a Volume, but with a lifecycle independent of any pod using the PV.

They can be created manually using a storageClass, PVC

A user can create PV directly by creating a PersistentVolumeClaim (PVC) configured with the StorageClass. The PV is created after the PVC is fulfilled by the storageclass provisioner, so think of the PVC as a request for storage and the PV as fulfillment.

or automatically (dynamically), also with storageClass

If they are created by the system through a chart/deploy/replicaset using the StorageClass and provisioner then it is considered automatic/dynamic. Resources that need storage will use the default StorageClass if one is set without any prompting.

Storage volume providers/provisioners

These will be used mostly with persistent volumes, though they could also be used for regular volumes. If the volume does not need to be persistent however then they don’t provide any advantage and something like a hostpath should be fine.

Cloud storage provisioners

Cloud storage drivers provide storage completely separate from local container or host from a cloud service. GKE, Amazon, DigitalOcean, Azure are some possible providers, there are others.

Storage classes define resources to create storage -

Storageclass defines a provisioner which does the actual storage creation and handling. The storageClassName is used to request storage from that provisioner.

The persistentVolumeReclaimPolicy controls how PV deletion is handled

PVs stick around until they are deleted. The persistentVolumeReclaimPolicy: determines exactly what deleting them means however.
Delete - When the PVC is deleted, the volume it is associated with is also deleted.
Retain - When PVC is deleted, the volume is kept, so the data is preserved and it could be used elsewhere.

Creating PV using PersistentVolumeClaim

This creates a new PersistentVolume from the configured storage class. This example is digitalocean, but it should just be a different storageClassName for other providers.

apiVersion: v1  
kind: PersistentVolumeClaim  
metadata:  
  name: docsi-pvc-test  
spec:  
  accessModes:  
  - ReadWriteOnce  
  resources:  
    requests:  
      storage: 10Gi  
  storageClassName: do-block-storage  

The reclaim policy is not configurable in a PVC
The storageClass determines the default policy, so a modified copy would be needed to change the default.
Patching a pv so the volume reclaim policy it retain, not delete -
If I need a pv to have a retain policy, I just patch it.

kubectl patch pv pvc-d8f04adb-c46b-11e9-af75-6e9f2a8237eb -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'  

Example of mounting a persistent volume in a deploy -

In this example the volume was created separately.
This is under spec.template.spec.containers -

  volumeMounts:  
   - mountPath: /data/example  
     name: persistent-psdata  

And this is under spec.template.spec.volumes -
Note that the name is local to the deployment, but the claimName must match the pvc you want to attach.

   - name: persistent-example  
     persistentVolumeClaim:  
       claimName: docsi-pvc-test  

Deleting a pvc -

Have to make sure the pvc is not protected; if first command indicates pvc-protection, it is and you need to remove that protection.

kubectl describe pvc csi-pvc-test | grep Finalizers  

This command can then be used to remove protection.

kubectl patch pvc csi-pvc-test -p '{"metadata":{"finalizers": []}}' --type=merge  

Then delete it using kubectl delete -

kubectl delete pvc csi-pvc-test  

Example using a hostpath volume

A couple quick examples to access files relevant to a specific host. This gives a container access to a socket on the host, and directory to write data to that will be host specific. These are not persistent, but in these cases that works fine.
Set up hostPaths for a directory, file -

      - name: dockersock  
        hostPath:  
          path: /var/run/docker.sock  
      - name: data  
        hostPath:  
          path: /var/lib/filebeat-data  
          type: DirectoryOrCreate  

Then mount them to be used directly in the containers -

        volumeMounts:  
        - name: data  
          mountPath: /usr/share/filebeat/data  
        - name: dockersock  
          mountPath: "/var/run/docker.sock"  

Volumesnapshot -

Point in time image of volumes. Can use k8s objects or cloud provider tool like doctl -
The object type is VolumeSnapshot. With digitalocean using doctl tool it is:

doctl compute volume snapshot <volume-id> --snapshot-name "Give-it-a-name"   

Set up the volumesnapshot

The pvc name is used to select the volume for the snapshot. Make sure to configure the namespace (if you are using them) or the pvc will not be found.
snapshot.yml:

apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
  name: csi-do-test-snapshot
  namespace: test
spec:
  source:
    name: docsi-pvc-test
    namespace: test
    kind: PersistentVolumeClaim

Do the snapshot -

kubectl apply -f snapshot.yml  

Restore from the snapshot

A pvc is created using the snapshot as a source. Size needs to be same or greater than original.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-master-restore
  namespace: test
spec:
  dataSource:
    name: csi-do-test-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Reconnecting a persistent volume that was retained

This applies to a persistent volume that had a retain policy. It only applied when I deleted a cluster but wanted to use the persistent volume in another cluster.
Reclaim the volume
In order to reclaim the volume, I needed to create a PersistentVolume, but with some atypical configuration. This is digitalocean specific. The volumehandle was needed to get the persistent volume to attach the specific volume you are targeting for re-use.
I found this value using doctl command:

doctl compute volume list  

Then create the PersistentVolume:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-test-retrieve
  annotations:
    # fake it by indicating this is provisioned dynamically
    pv.kubernetes.io/provisioned-by: dobs.csi.digitalocean.com
spec:
  storageClassName: do-block-storage
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  csi:
    driver: dobs.csi.digitalocean.com
    fsType: ext4
    volumeHandle: d9745c38-c46b-11e9-8b09-0a58ac14e056
    volumeAttributes:
      com.digitalocean.csi/noformat: "true"

Then create a PVC that points to the PV you just set up

In this case the creation order is backward, set up the PV, then do a standard PVC that targets the name you gave to the PV, pv-test-retreive in this case.