If you are working with Kubernetes, it’s pretty important to be able to generate variations of your configuration. Your production cluster probably has TLS settings that won’t make sense in while testing locally in Minikube, or you’ll have service types that should be NodePort here and LoadBalancer there.
While tools like Helm tackle this problem with PHP style templates, kustomize offers a different approach based on constructing config via composition.
With each piece of Kubernetes config uniquely identified, composite key style, through a combination of kind, apiVersion and metadata.name, kustomize can generate new config by patching one yaml with others.
This lets us generate config without wondering what command line arguments a given piece of config might have been created from, or having turing complete programming languages embedded in it.
This sounds a little stranger than it is, so let’s make this more concrete with an example.
Lets say you clone a project and see kustomization.yaml files lurking in some of the subdirectories.
mike@sleepycat:~/projects/hello_world$ tree
.
├── base
│ ├── helloworld-deployment.yaml
│ ├── helloworld-service.yaml
│ └── kustomization.yaml
└── overlays
├── gke
│ ├── helloworld-service.yaml
│ └── kustomization.yaml
└── minikube
├── helloworld-service.yaml
└── kustomization.yaml
4 directories, 7 files
This project is using kustomize. The folder structure suggests that there is some base configuration, and variations for GKE and Minikube. We can generate the Minikube version with kubectl kustomize overlays/minikube.
This actually shows one of the nice things about kustomize, you probably already have it, since it was built into the kubectl command in version 1.14 after a brief kerfuffle.
mike@sleepycat:~/projects/hello_world$ kubectl kustomize overlays/minikube/
apiVersion: v1
kind: Service
metadata:
labels:
app: helloworld
name: helloworld
spec:
ports:
- name: "3000"
port: 3000
protocol: TCP
targetPort: 3000
selector:
app: helloworld
type: NodePort
status:
loadBalancer: {}
...more yaml...
If you want to get this config into your GKE cluster, it would be as simple as kubectl apply -k overlays/gke.
This tiny example obscures one of the other benefits of kustomize: it sorts the configuration it outputs in the following order to avoid dependency problems:
- Namespace
- StorageClass
- CustomResourceDefinition
- MutatingWebhookConfiguration
- ServiceAccount
- PodSecurityPolicy
- Role
- ClusterRole
- RoleBinding
- ClusterRoleBinding
- ConfigMap
- Secret
- Service
- LimitRange
- Deployment
- StatefulSet
- CronJob
- PodDisruptionBudget
Because it sorts it’s output this way, kustomize makes it far less error prone to get your application up and running.
Setting up your project to use kustomize
To get your project set up with kustomize, you will want a little more than the functionality built into kubectl. There are a few ways to install kustomize, put I think the easiest (assuming you have Go on your system) is go get:
go get sigs.k8s.io/kustomize
With that installed, we can create some folders and use the fd command to give us the lay of the land.
$ mkdir -p {base,overlays/{gke,minikube}}
$ fd
base
overlays
overlays/gke
overlays/minikube
In the base folder we’ll need to create a kustomization file some config. Then we tell kustomize to add the config as resources to be patched.
base$ touch kustomization.yaml base$ kubectl create deployment helloworld --image=mikewilliamson/helloworld --dry-run -o yaml > helloworld-deployment.yaml base$ kubectl create service clusterip helloworld --tcp=3000 --dry-run -o yaml > helloworld-service.yaml base$ kustomize edit add resource helloworld-*
The kustomize edit series of commands (add, fix, remove, set) all exist to modify the kustomization.yaml file.
You can see that kustomize edit add resource helloworld-* added a resources: key with an array of explicit references rather than an implicit file glob.
$ cat base/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - helloworld-deployment.yaml - helloworld-service.yaml
Moving over to the overlays/minikube folder we can do something similar.
minikube$ touch kustomization.yaml minikube$ kubectl create service nodeport helloworld --tcp=3000 --dry-run -o yaml > helloworld-service.yaml minikube$ kustomize edit add patch helloworld-service.yaml minikube$ kustomize edit add base ../../base
Worth noting is the base folder where kustomize will look for the bases to apply the patches to. The resulting kustomization.yaml file looks like the following:
$ cat overlays/minikube/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization patchesStrategicMerge: - helloworld-service.yaml bases: - ../../base
One final jump into the overlays/gke folder gives us everything we will need to see the difference between two configs.
gke$ kubectl create service loadbalancer helloworld --tcp=3000 --dry-run -o yaml > helloworld-service.yaml gke$ touch kustomization.yaml gke$ kustomize edit add base ../../base gke$ kustomize edit add patch helloworld-service.yaml
Finally we can generate the two different configs and diff them to see the changes.
$ diff -u --color <(kustomize build overlays/gke) <(kustomize build overlays/minikube/)
--- /dev/fd/63 2019-05-31 21:38:56.040572159 -0400
+++ /dev/fd/62 2019-05-31 21:38:56.041572186 -0400
@@ -12,7 +12,7 @@
targetPort: 3000
selector:
app: helloworld
- type: LoadBalancer
+ type: NodePort
status:
loadBalancer: {}
---
It won't surprise you that what you see here is just scratching the surface. There are many more fields that are possible in a kustomization.yaml file, and nuance in what should go in which file given that kustomize only allows addition not removal.
The approach kustomize is pursuing feels really novel in a field that has been dominated by DSLs (which hide the underlying construct) and templating (with the dangers of embedded languages and concatenating strings).
Working this way really helps deliver on the promise of portability made by Kubernetes; Thanks to kustomize, you’re only a few files and a `kustomize build` away from replatforming if you need to.
mike.williamson 