Testing an OpenTelemetry Collector deployed as a Daemonset in Kubernetes

TL;DR: skip to this readme for the easiest path

When you’re running applications in Kubernetes, and you’re using OpenTelemetry for observability, then one recommended practice is to run the OpenTelemetry Collector on each node in your cluster. Then each of your applications can send telemetry data (traces, metrics, logs) to that local collector.

To run on each node, the collector runs as a daemonset (instead of a deployment). It is not a Kubernetes service, so it does not get a name in cluster DNS. Instead, it lives at $NODE_IP, on whatever ports it opens (4318 for OTLP over HTTP). Where does $NODE_IP come from? Ya gotta pass hostIP into a pod in its definition.

I followed this blog post to set up my collector daemonset. Now how do I check whether it’s working??

It isn’t exposed outside the cluster, so I can’t test with curl from my local machine. I need to run the test from a pod within the cluster, a pod that has $NODE_IP available to it, a pod with the necessary tools installed to test a span.

Spin up a pod inside the cluster

To get that $NODE_IP environment variable, I need yaml. Here is a yaml that’ll spin up an ubuntu image; put it in a file called otel-test-pod.yaml :

apiVersion: v1
kind: Pod
metadata:
  labels:
    why: test-the-opentelemetry-collector
  name: otel-test-pod
spec:
  containers:
    - args:
        - sleep
        - "3600"
      image: ubuntu:22.04
      name: otel-test-pod
      resources: {}
      env:
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
  dnsPolicy: ClusterFirst
  restartPolicy: Never

Apply the yaml to start the pod:

kubectl apply -f otel-test-pod.yaml

Shell into the pod:

kubectl exec otel-test-pod -i --tty -- bash

Install the tools you need:

apt update
apt-get install curl

Send a test span

Next, you’ll need some data to send. Create a test span in a file called sample-span.json

	{
	"resourceSpans": [
	{
	"resource": {
	"attributes": [
	{
	"key": "service.name",
	"value": {
	"stringValue": "otel-test-pod"
	}
	}
	]
	},
	"scopeSpans": [
	{
	"scope": {
	"name": "manual-test"
	},
	"spans": [
	{
	"traceId": "71699b6fe85982c7c8995ea3d9c95df2",
	"spanId": "3c191d03fa8be065",
	"name": "test span",
	"kind": 2,
	"droppedAttributesCount": 0,
	"events": [],
	"droppedEventsCount": 0,
	"status": {
	"code": 1
	}
	}
	]
	}
	]
	}
	]
	}

view raw sample-span.json hosted with ❤ by GitHub

You could send this directly, but that test only works once. Here’s a command that sends it, but first substitutes in.a unique trace ID and Span ID:

cat sample-span.json | \
  sed "s/71699b6fe85982c7c8995ea3d9c95df2/$(openssl rand -hex 16)/" | \
  sed "s/3c191d03fa8be065/$(openssl rand -hex 8)/" | \
  curl -v $NODE_IP:4318/v1/traces -H 'Content-Type: application/json' -d @-

If this works, you’ll see a response that says {} or {"partialSuccess":{}}.

My collector didn’t show anything in the logs when I did this, even though I have logs at debug. I did find the span in Honeycomb, so it worked. If yours didn’t work, check the collector logs for errors.

Delete the pod

Clean up!

kubectl delete pod otel-test-pod

Can we make this easier?

A little bit. I made a docker image that contains that sample-span.json and a test script that sends it.

Here’s the repo, with the easier instructions in the README. The image is on DockerHub; it’s ubuntu + curl + sample-span.json + a bash script.

Seeing inside a kubernetes cluster is tricky. This is exactly what observability helps with. Getting that set up is an exercise in prying it open sometimes!