How do I update the Observe K8s Agent "TRACE_TOKEN" to resolve Unauthenticated 401 errors?

Data Ingestion
, ,
1

When monitoring Kubernetes clusters with the Observe K8s Agent, you might encounter an error indicating that the agent is unable to export trace data due to an authentication issue. This article explains how to resolve the specific error shown below by updating the TRACE_TOKEN in the Observe K8s Agent’s configuration.

If you see the following error this indicates that the Observe K8s Agent failed to authenticate with the Observe data collection endpoint:

2025-03-13T19:20:45.575Z    error    internal/queue_sender.go:105    Exporting failed. Dropping data.    {“kind”: “exporter”, “data_type”: “traces”, “name”: “otlphttp/observe/forward/trace”, “error”: “not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://<CUSTOMER_ID>.collect.observeinc.com/v2/otel/v1/traces responded with HTTP Status Code 401”, “dropped_items”: 9}

This error typically occurs when the TRACE_TOKEN, the credential used by the Observe agent to authenticate with the Observe platform, is either missing, expired, or incorrect.

Solution

To resolve the authentication error, you need to update the TRACE_TOKEN stored in the agent’s secret and restart the agent. Follow these steps:

Step 1: Update the TRACE_TOKEN in the Secret:

The TRACE_TOKEN is stored in a Kubernetes secret named agent-credentials within the observe namespace. You’ll use kubectl patch to update this secret with the new token.

Run the following command, replacing <YOUR_TOKEN> with your actual TRACE_TOKEN:

$ kubectl patch secret agent-credentials -n observe --type=‘merge’ \   -p “{\“data\”:{\“TRACE_TOKEN\”:\”$(echo -n ‘<YOUR_TOKEN>’ | base64)\“}}”

Step 2: Restart the Observe Agent:

After updating the secret, restart the observe-agent-forwarder-agent DaemonSet to apply the changes:

$ kubectl rollout restart daemonset observe-agent-forwarder-agent --namespace observe

To view the updated agent-credentials secret:

$ kubectl get secret agent-credentials -n observe -o yaml

Validate you are no longer seeing the Unauthenticated 401 errors and that your TRACE events are ingested in your Observe tenant.