k3s on arm64

Jun 19, 2019 - kubernetes k3s arm64 rock64

I’m evaluating k3s a Lightweight Kubernetes on a 3 nodes arm64 cluster (RK3328 Quad arm64).

At the time of writing the stable release is k3s v0.6.1.

Here are my notes:

If you haven’t installed k3s with the install.sh, you may need to load some modules: br_netfilter and overlay
Docker is not needed since k3s is using containerd but it seems I had to start docker to initialized the whole cgroups, at least on Arch

Remember to modify all the templates to use an arm64 image

            containers:
            - name: kubernetes-dashboard
              image: k8s.gcr.io/kubernetes-dashboard-arm64:v1.10.1

Deploy metrics server Clone the k3s and change the image to be arm64 k8s.gcr.io/metrics-server-arm64:v0.3.1

     git clone git@github.com:rancher/k3s.git
     vi k3s/recipes/metrics-server/metrics-server-deployment.yaml
     sudo  k3s kubectl -n kube-system create -f k3s/recipes/metrics-server

Test it by running k3s kubectl top node

Dynamic storage class
Kubernetes provides a local storage but no dynamic volume provisioning, you still have to manually create the PersistentVolume which breaks any automatic deployment (especially from Helm).
Rancher developed local-path-provisioner for that purpose. I couldn’t find a recent build for arm64, so I built one, you can find it here.

curl -sfL https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml > local-path-storage.yaml
# Edit local-path-storage.yaml to set image to akhenakh/local-path-provisioner:arm64-v0.0.9
kubectl create -f local-path-storage.yaml
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Deploy dashboard
Change the image to be arm64 k8s.gcr.io/kubernetes-dashboard-arm64:v1.10.1.
Add the skip option to the args - --enable-skip-login

        curl -sfL https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml > kubernetes-dashboard.yaml
        vi kubernetes-dashboard.yaml
        cat <<EOF | sudo k3s kubectl create -f -
        apiVersion: rbac.authorization.k8s.io/v1beta1
        kind: ClusterRoleBinding
        metadata:
          name: kubernetes-dashboard
          labels:
            k8s-app: kubernetes-dashboard
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
          name: cluster-admin
        subjects:
        - kind: ServiceAccount
          name: kubernetes-dashboard
          namespace: kube-system
        EOF
        sudo  k3s kubectl -n kube-system create -f kubernetes-dashboard.yaml

Merge /etc/rancher/k3s/k3s.yaml to your .kube/config on your workstation host, then kubectl proxy to localhost:8001

Install a different version

    curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.6.1 sh -

Install agents

    curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.6.1 K3S_URL=https://mymaster.home:6443 K3S_TOKEN=K101864xxxxxxxxxxxxxxxxxxx::node:xxxxxxxxxxxxxxxxxxxxxxxxx sh -

See Also