So far, Percona Kubernetes Operator for Percona Server for MongoDB (PSMDB) has supported only managing replica sets, but from version 1.6.0 it is possible to start a sharding cluster, although at the moment only with one shard. This is a step toward supporting full sharding, with multiple shards being added in a future release.
Components that were added to make this work are config replica set and mongos support, with all things that go around that like services, probes, statuses, etc. As well as starting a sharded cluster from scratch, it is also possible to migrate from a single replica set to a shard setup – and back.
Configuration Options for Sharding
A new section was added into the cr.yaml configuration called “sharding” where you can enable/disable sharding altogether. You can also change the number of running pods for config server replica set and mongos, set antiAffinityTopologyKey, podDisruptionBudget, resources, and define how the mongos service will be exposed.
Here’s how some simple config might look like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | sharding: enabled: true configsvrReplSet: size: 3 volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi mongos: size: 3 affinity: antiAffinityTopologyKey: "kubernetes.io/hostname" podDisruptionBudget: maxUnavailable: 1 resources: limits: cpu: "300m" memory: "0.5G" requests: cpu: "300m" memory: "0.5G" expose: enabled: true exposeType: LoadBalancer |
The default number of pods for config server replica set and mongos is three, but you can use less if you enable the “allowUnsafeConfigurations” option.
There are more configuration options inside the cr.yaml, but some of them are commented out since they are probably a bit more specific to different use cases or environments.
This is how the pods and service setup might look like when you start the sharding cluster:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | NAME READY STATUS RESTARTS AGE my-cluster-name-cfg-0 2/2 Running 0 2m38s my-cluster-name-cfg-1 2/2 Running 1 2m10s my-cluster-name-cfg-2 2/2 Running 1 103s my-cluster-name-mongos-556bdd5b79-bkgd2 1/1 Running 0 2m36s my-cluster-name-mongos-556bdd5b79-klkh6 1/1 Running 0 2m36s my-cluster-name-mongos-556bdd5b79-nbgd9 1/1 Running 0 2m36s my-cluster-name-rs0-0 2/2 Running 0 2m40s my-cluster-name-rs0-1 2/2 Running 1 2m11s my-cluster-name-rs0-2 2/2 Running 1 104s percona-server-mongodb-operator-587658ccc8-k6zpt 1/1 Running 0 3m14s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-cluster-name-cfg ClusterIP None <none> 27017/TCP 2m58s my-cluster-name-mongos LoadBalancer 10.51.244.4 34.78.50.13 27017:31685/TCP 2m56s my-cluster-name-rs0 ClusterIP None <none> 27017/TCP 3m |
Here you can see that in this example, we have mongos service configured to be exposed with LoadBalancer and available through external IP. At the current moment, the client will connect to mongos instances through the load balancer service in a round-robin fashion, but in the future, it is planned to support session affinity (sticky method) so that the same client would connect to the same mongos instance most of the time.
Migrating From Replica Set to One Shard Setup (and Back)
MongoDB (in general) supports migrating from replica set to sharding setup and also from sharding to replica set, but it requires more or less manual steps depending on the complexity of existing architecture. Our Kubernetes operator, at the current moment, supports automatic migration from replica set to one shard and back from one shard to replica set.
These are the steps that PSMDB Kubernetes Operator does when we enable sharding but have an existing replica set:
- restart existing replica set members with “–shardsvr” option included
- deploy config server replica set and mongos as they are defined in cr.yaml (default is three pods for each)
- create stateful set for config replica set
- setup Kubernetes service for mongos and config replica set
- add existing replica set as a shard in sharding cluster
In this process, data is preserved, but there might be additional steps needed with application users since they will become shard local users and not available through mongos (so it is needed to create them from mongos).
When we migrate from one shard setup to replica set, data is also preserved, the steps which are mentioned above are reverted, but in this case, application users are lost since they were stored in config replica set which doesn’t exist anymore – so they will need to be recreated.
SmartUpdate Strategy for Sharding Cluster
As you may know, both Percona Kubernetes Operators (Percona XtraDB Cluster and PSMDB) have SmartUpdate strategy which tries to upgrade the clusters automatically and with as little interruption for the application as possible.
When we are talking about sharding, this is what the steps look like:
- disable the balancer
- upgrade config replica set (secondaries first, step down primary, upgrade primary)
- upgrade data replica set (secondaries first, step down primary, upgrade primary)
- upgrade mongos pods
- enable balancer
This is how this process might look in the Operator logs when we upgrade the cluster from one PSMDB version to another (some parts stripped for brevity):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | {"level":"info","msg":"update Mongo version to 4.2.7-7 (fetched from db)"} {"level":"info","msg":"waiting for config RS update"} {"level":"info","msg":"statefullSet was changed, start smart update","name":"my-cluster-name-cfg"} {"level":"info","msg":"balancer disabled"} {"level":"info","msg":"primary pod is my-cluster-name-cfg-0.my-cluster-name-cfg.psmdb-test.svc.cluster.local:27017"} {"level":"info","msg":"apply changes to secondary pod my-cluster-name-cfg-2"} {"level":"info","msg":"pod my-cluster-name-cfg-2 started"} {"level":"info","msg":"apply changes to secondary pod my-cluster-name-cfg-1"} {"level":"info","msg":"pod my-cluster-name-cfg-1 started"} {"level":"info","msg":"doing step down..."} {"level":"info","msg":"apply changes to primary pod my-cluster-name-cfg-0"} {"level":"info","msg":"pod my-cluster-name-cfg-0 started"} {"level":"info","msg":"smart update finished for statefulset","statefulset":"my-cluster-name-cfg"} {"level":"info","msg":"statefullSet was changed, start smart update","name":"my-cluster-name-rs0"} {"level":"info","msg":"primary pod is my-cluster-name-rs0-0.my-cluster-name-rs0.psmdb-test.svc.cluster.local:27017"} {"level":"info","msg":"apply changes to secondary pod my-cluster-name-rs0-2"} {"level":"info","msg":"pod my-cluster-name-rs0-2 started"} {"level":"info","msg":"apply changes to secondary pod my-cluster-name-rs0-1"} {"level":"info","msg":"pod my-cluster-name-rs0-1 started"} {"level":"info","msg":"doing step down..."} {"level":"info","msg":"apply changes to primary pod my-cluster-name-rs0-0"} {"level":"info","msg":"pod my-cluster-name-rs0-0 started"} {"level":"info","msg":"smart update finished for statefulset","statefulset":"my-cluster-name-rs0"} {"level":"info","msg":"update Mongo version to 4.2.8-8 (fetched from db)"} {"level":"info","msg":"waiting for mongos update"} {"level":"info","msg":"balancer enabled"} |
Conclusion
Although adding support for one shard cluster doesn’t sound too important since it doesn’t allow sharding data across shards, it is a big milestone and laying the foundation for things that are needed in the future to support this. Except for that, it might allow you to expose your data to applications in different ways through mongos instances, so if interested please check the documentation and release notes for more details.
