More and more companies are adopting GitOps as the way of implementing Continuous Deployment. Its elegant approach built upon a well-known tool wins the hearts of engineers. But even if your git repository is private, it’s strongly discouraged to store keys and passwords in unencrypted form.
This blog post will show how easy it is to use GitOps and keep Kubernetes secrets for Percona Kubernetes Operator for Percona Server for MongoDB securely in the repository with Sealed Secrets or Vault Secrets Operator.
Sealed Secrets
Prerequisites:
- Kubernetes cluster up and running
- Github repository (optional)
- The files and keys that I used are stored in a public blog-data repository
Install Sealed Secrets Controller
Sealed Secrets rely on asymmetric cryptography (which is also used in TLS), where the private key (which in our case is stored in Kubernetes) can decrypt the message encrypted with the public key (which can be stored in public git repository safely). To make this task easier, Sealed Secrets provides the kubeseal tool, which helps with the encryption of the secrets.
Install kubeseal operator into your Kubernetes cluster:
1 | kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.15.0/controller.yaml |
It will install the controller into the kube-system namespace and provide the Custom Resource Definition sealedsecrets.bitnami.com. All resources in Kubernetes with kind: SealedSecretswill be handled by this Operator.
Download the kubeseal binary:
1 2 | wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.15.0/kubeseal-linux-amd64 -O kubeseal sudo install -m 755 kubeseal /usr/local/bin/kubeseal |
Encrypt the Keys
In this example, I intend to store important secrets of the Percona Kubernetes Operator for Percona Server for MongoDB in git along with my manifests that are used to deploy the database.
First, I will seal the secret file with system users, which is used by the MongoDB Operator to manage the database. Normally it is stored in deploy/secrets.yaml.
1 | kubeseal --format yaml < secrets.yaml > blog-data/sealed-secrets/mongod-secrets.yaml |
This command creates the file with encrypted contents, you can see it in the blog-data/sealed-secrets repository here. It is safe to store it publicly as it can only be decrypted with a private key.
Executing kubectl apply -f blog-data/sealed-secrets/mongod-secrets.yaml does the following:
- A sealedsecrets custom resource (CR) is created. You can see it by executing kubectl get sealedsecrets.
- The Sealed Secrets Operator receives the event that a new sealedsecrets CR is there and decrypts it with the private key.
- Once decrypted, a regular Secrets object is created which can be used as usual.
1 2 3 4 5 6 7 | $ kubectl get sealedsecrets NAME AGE my-secure-secret 20m $ kubectl get secrets my-secure-secret NAME TYPE DATA AGE my-secure-secret Opaque 10 20m |
Next, I will also seal the keys for my S3 bucket that I plan to use to store backups of my MongoDB database:
1 2 | kubeseal --format yaml < backup-s3.yaml > blog-data/sealed-secrets/s3-secrets.yaml kubectl apply -f blog-data/sealed-secrets/s3-secrets.yaml |
Vault Secrets Operator
Sealed Secrets is the simplest approach, but it is possible to achieve the same result with HashiCorp Vault and Vault Secrets Operator. It is a more advanced, mature, and feature-rich approach.
Prerequisites:
- Kubernetes cluster up and running
- Github repository (optional)
- The files and keys that I used are stored in a public blog-data repository
- HashiCorp Vault up and running
Vault Secrets Operator also relies on Custom Resource, but all the keys are stored in HashiCorp Vault:
Preparation
Create a policy on the Vault for the Operator:
1 2 3 4 5 | cat <<EOF | vault policy write vault-secrets-operator - path "kvv2/data/*" { capabilities = ["read"] } EOF |
The policy might look a bit differently, depending on where your secrets are.
Create and fetch the token for the policy:
$ vault token create -period=24h -policy=vault-secrets-operator
1 2 3 4 | Key Value --- ----- token s.0yJZfCsjFq75GiVyKiZgYVOm ... |
Write down the token, as you will need it in the next step.
Create the Kubernetes Secret so that the Operator can authenticate with the Vault:
1 2 3 4 5 6 7 8 9 10 11 12 13 | export VAULT_TOKEN=s.0yJZfCsjFq75GiVyKiZgYVOm export VAULT_TOKEN_LEASE_DURATION=86400 cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Secret metadata: name: vault-secrets-operator type: Opaque data: VAULT_TOKEN: $(echo -n "$VAULT_TOKEN" | base64) VAULT_TOKEN_LEASE_DURATION: $(echo -n "$VAULT_TOKEN_LEASE_DURATION" | base64) EOF |
Deploy Vault Secrets Operator
It is recommended to deploy the Operator with Helm, but before we need to create the values.yaml file to configure the operator.
1 2 3 4 5 6 7 8 9 10 11 12 13 | environmentVars: - name: VAULT_TOKEN valueFrom: secretKeyRef: name: vault-secrets-operator key: VAULT_TOKEN - name: VAULT_TOKEN_LEASE_DURATION valueFrom: secretKeyRef: name: vault-secrets-operator key: VAULT_TOKEN_LEASE_DURATION vault: address: "http://vault.vault.svc:8200" |
Environment variables are pointing to the Secret that was created in the previous chapter to authenticate with Vault. We also need to provide the Vault address for the Operator to retrieve the secrets.
Now we can deploy the Vault Secrets Operator:
1 2 3 4 | helm repo add ricoberger https://ricoberger.github.io/helm-charts helm repo update helm upgrade --install vault-secrets-operator ricoberger/vault-secrets-operator -f blog-data/sealed-secrets/values.yaml |
Give me the Secret
I have a key created in my HashiCorp Vault:
1 2 3 4 5 6 7 8 9 10 11 | $ vault kv get kvv2/mongod-secret … Key Value --- ----- MONGODB_BACKUP_PASSWORD <> MONGODB_CLUSTER_ADMIN_PASSWORD <> MONGODB_CLUSTER_ADMIN_USER <> MONGODB_CLUSTER_MONITOR_PASSWORD <> MONGODB_CLUSTER_MONITOR_USER <> MONGODB_USER_ADMIN_PASSWORD <> MONGODB_USER_ADMIN_USER <> |
It is time to create the secret out of it. First, we will create the Custom Resource object of kind: VaultSecret.
1 2 3 4 5 6 7 8 9 10 | $ cat blog-data/sealed-secrets/vs.yaml apiVersion: ricoberger.de/v1alpha1 kind: VaultSecret metadata: name: my-secure-secret spec: path: kvv2/mongod-secret type: Opaque $ kubectl apply -f blog-data/sealed-secrets/vs.yaml |
The Operator will connect to HashiCorp Vault and create regular Secret object automatically:
1 2 3 4 5 6 7 | $ kubectl get vaultsecret NAME SUCCEEDED REASON MESSAGE LAST TRANSITION AGE my-secure-secret True Created Secret was created 47m 47m $ kubectl get secret my-secure-secret NAME TYPE DATA AGE my-secure-secret Opaque 7 47m |
Deploy MongoDB Cluster
Now that the secrets are in place, it is time to deploy the Operator and the DB cluster:
1 2 | kubectl apply -f blog-data/sealed-secrets/bundle.yaml kubectl apply -f blog-data/sealed-secrets/cr.yaml |
The cluster will be up in a minute or two and use secrets we deployed.
By the way, my cr.yaml deploys MongoDB cluster with two shards. Multiple shards support was added in version 1.7.0of the Operator – I encourage you to try it out. Learn more about it here: Percona Server for MongoDB Sharding.
