Using OpenTelemetry in Kubernetes

Understanding Metrics in OpenTelemetry (OTel) API
FusionReactor vs AppDynamics

Using OTel auto-instrumentation/agents – OpenTelemetry in Kubernetes

Maybe you heard about Opentelemetry, Kubernetes, or Opentelemetry in Kubernetes, and you don’t know what it is, or you want to learn more? This article will discuss how the OpenTelemetry (OTel) new collector feature simplifies workloads deployment on Kubernetes.

Opentelemetry is an open-source project hosted by the Cloud Native Computing Foundation (CNCF) that provides a standard way to generate telemetry data. OpenTelemetry bridges the gap between service providers and users alike when collecting data from a diverse set of systems. This helps providers and users gain a broad picture of the application’s performance and the underlying reasons for the result.

As the adoption of Cloud-native ecosystems continues to grow, the complexities of distributed systems and microservices will also expand, resulting in the need for a simple and scalable way to deploy workloads on Kubernetes.

One of the most tedious processes when deploying an observability solution is instrumentation. Presently, there are two approaches to instrument an application: manual/explicit and automatic instrumentation.

Manual/explicit: Developers can use pre-built instrumentation libraries or Opentelemetry APIs to instrument the application’s source code.

Automatic: the entire instrumentation process is on automation without any code modification and recompilation

The dawn of Opentelemetry changes a lot when automating instrumentation. What used to be a proprietary technology delivered by several Application Performance Monitoring (APM) or observability vendors, users can enjoy vendor-neutral instrumentation as an open-source technology.

Regardless of the possibility of using Opentelemtry as an open-source technology, deploying Opentelemtry auto-instrumentation at scale or validating proof of value on Kubernetes can still be a major problem. It’s hard to ignore the increasing number of immutable containers in modern applications. This necessitates that adding instrumentation to existing images requires rebuilding containers, resulting in a tedious and costly project. Using Opentelemtry Operator’s new features on Kubernetes solves this problem.

Instrumentation CR in OTel operator used in OpenTelemetry in Kubernetes

The new auto-instrumentation agent, OpenTelemetry 0.38.0, introduced a power-packed feature known as Instrumentation custom resource (CR). This feature defines the configuration for OpenTelemtry SDK. Instrumentation CR presence in the cluster and namespace of workload annotation enables SDK instrumentation in a simplified manner.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
python:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
EOF.

Presently, the instrumentation feature is supported across different languages such as Java, NodeJS, and Python once the workload or namespace is presented with the following annotation for different languages.

  • instrumentation.opentelemetry.io/inject-java: “true” — for Java
  • instrumentation.opentelemetry.io/inject-nodejs: “true” — for NodeJS
  • instrumentation.opentelemetry.io/inject-python: “true” — for Python

From here, the operator can introduce OpenTelemetry auto-instrumentation libraries into the application container and then configure the instrumentation so the auto-instrumentation agent can export data to an endpoint in the instrumentation CR.

Java example with Spring Pet clinic

Looking at this from another perspective, let’s deploy the Spring Petclinic Java application with Opentelemetry collector in mind. The application will be instrumented and data communicated to an OpenTelemetry collector.

First, let’s create the following deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
name: spring-petclinic
spec:
selector:
matchLabels:
app: spring-petclinic
replicas: 1
template:
metadata:
labels:
app: spring-petclinic
annotations:
sidecar.opentelemetry.io/inject: "true"
instrumentation.opentelemetry.io/inject-java: "true"
spec:
containers:
- name: app
image: ghcr.io/pavolloffay/spring-petclinic:latest
and apply the instrumentation annotation:
kubectl patch deployment.apps/spring-petclinic -p '{"spec": {"template": {"metadata": {"annotations": {"instrumentation.opentelemetry.io/inject-java": "true"}}}}}'

Something unique happens when the instrumentation annotation is applied. The spring-petclinic pod restarts its operation and the newly started pod will be instrumented with OpenTelemtry Java auto-instrumentation. OTLP reports the telemetry data to the collector at http://otel-collector:4317.
We have attached the following code snippet to guide you through the deployment process.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
spec:
config: |
receivers:
otlp:
protocols:
grpc:
HTTP:
processors:

exporters:
logging:

service:
pipelines:
traces:
receivers: [otlp]
processors: []
exporters: [logging]
EOF

If you look closely, you can see that we can port-forward the application HTTP port via kubectl port-forward deployment.apps/spring-pet clinic 8080:8080 and review the application on the WEB. The OpenTelemetry collector can view the data spans via kubectl logs deployment.apps/otel-collector.

Application Instrumentation can be a complicated process depending on the language applied. For instance, when we look at Java, the auto-instrumentation is called Java agent, and it does bytecode manipulation. This procedure injects instrumentation points to unique code paths. After that, the points create telemetry data when the final code is executed.

How is the injection logic implemented?

The process might sound tricky, but let me explain using Java agent, although similar standards for other runtimes. The OpenTelemetry operator implements a special feature called mutating admission webhook. This feature is invoked during the creation of the Pod object, and during the process, the webhook modifies the Pod object to introduce auto-instrumentation libraries into the application container.

The Javaagent is introduced into the application container through an init container that copies that Javaagent into a volume mounted to the application container. Configuring the SDK is done by introducing environment variables.
Into the application container.

The final step is to configure the JVM to utilize the auto-instrumentation agent. The Javaagent can do this by configuring the environment variable to utilize the Javaagent.

Conclusion – OpenTelemetry in Kubernetes

In conclusion, we expect to see continued investment in Kubernetes, especially in workload deployment. OpenTelemetry auto-instrumentation on Kubernetes can simplify workload deployment with the operator pattern. There is no need to change the application container, resulting in a high value for a scalable way to transmit telemetry solutions. Presently, only Java, Python, and NodeJS runtimes are supported; hopefully, other languages will continue to gain momentum.