Volume Expansion, a feature that became generally available since Kubernetes version 1.24, allows users to increase the capacity of their Persistent Volumes and underlying storage within Kubernetes. There is no need to use clouds’ UI or APIs to do that anymore.
In our Operators, we use Stateful Sets – a Kubernetes resource that was designed for stateful workloads. For various reasons, Stateful Sets are mostly immutable. For instance, it is not possible to change the storage size (see this issue from 2018, still Open!). This makes it cumbersome to expand the volume for the applications that use Stateful Sets.
Percona Operator for MySQL, starting from version 1.14, supports automated volume expansion. This allows users to avoid manual work and human error. The feature is currently in technical preview.
How to use it
Before you begin, please check that your Kubernetes is newer than version 1.24 and that your Container Storage Interface (CSI) supports automated expansion by running the following command:
1 2 3 | % kubectl get storageclass <YOUR_STORAGE_CLASS> -o yaml | grep allowVolumeExpansion allowVolumeExpansion: true |
To increase the storage capacity, you just need to change the Percona XtraDBCluster custom resource. For example, my 00-cr.yaml has the following under storage section:
1 2 3 4 5 6 7 8 | pxc: size: 3 image: percona/percona-xtradb-cluster:8.0.35-27.1 volumeSpec: persistentVolumeClaim: resources: requests: storage: 6G |
To increase the storage size from six gigabytes, I just need to edit the custom resource or apply the new one. In my new 01-cr-resize.yaml I set 10 gigabytes of storage:
1 2 3 4 5 6 7 8 9 10 11 12 13 | % diff -u 00-cr.yaml 01-cr-resize.yaml --- 00-cr.yaml 2024-03-04 23:35:36 +++ 01-cr-resize.yaml 2024-03-04 23:50:11 @@ -16,7 +16,7 @@ persistentVolumeClaim: resources: requests: - storage: 6G + storage: 10G haproxy: enabled: true size: 2 |
So the way to move forward is to just apply the new manifest:
1 | kubectl apply -f 01-cr-resize.yaml |
The Operator will trigger the volume expansion by modifying the Persistent Volume Claims. In a minute or two, your volumes will be of a new size:
1 2 3 4 | NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE datadir-pvc-test-pxc-0 Bound pvc-12218c03-9009-48f2-94b0-2b07b91d97c0 10Gi RWO standard-rwo 19m datadir-pvc-test-pxc-1 Bound pvc-d7b44e99-094f-47d1-a960-ef82a9405dcd 10Gi RWO standard-rwo 17m datadir-pvc-test-pxc-2 Bound pvc-29a963fe-7c29-44b8-bb3d-aee9f20533c1 10Gi RWO standard-rwo 16m |
Volume expansion is completely transparent for the application and the database, as Pods are not restarted. You might notice, though, that the StatefulSet was recreated.
1 2 3 4 | % kubectl get sts NAME READY AGE pvc-test-pxc 3/3 23s |
It is happening because of the immutable nature of it. To solve this, Operator expands the volumes, and deletes and recreates the StatefulSet with the new volume size.
Limitations
Due to the limited capabilities of the storage and filesystems, it is only possible to increase the storage capacity but not shrink it. To reduce the storage footprint, you will need to take the manual route of creating new Pods with smaller volumes and syncing the data there. The steps are described in this blog post Change Storage Class on Kubernetes on the Fly (even if it is about another topic, the steps are the same).
Glimpse into the future
We already have automated volume expansion in the Percona Operator for PostgreSQL, next step will be to add this functionality into Percona Operator for MongoDB in the upcoming release Q1/Q2 2024. But it is just a first step towards complete storage autoscaling. The idea is to detect if storage reaches certain thresholds (capacity, IOPs, etc), and if it does – automatically scale the storage. We expect this functionality to be available for all Operators this year.
The Percona Kubernetes Operators lets you easily create and manage highly available, enterprise-ready MySQL, PostgreSQL, and MongoDB clusters on Kubernetes. Experience hassle-free database management and provisioning without the need for manual maintenance or custom in-house scripts.
Learn More About Percona Kubernetes Operators
