Observe-Agent Deploying Tip While deploying using Gitops Tools

Data Ingestion
, ,
1

Important: Environment Variable Overrides in Observe Agent Deployments

When deploying the Observe Agent through automation tools such as GitOps, ArgoCD, Spinnaker, or CI/CD pipelines, pay close attention to how environment variables (extraEnvs) are defined.

:light_bulb: Why This Matters

Kubernetes merges environment variables defined in the container spec in order, but Helm (and most templating tools) treats extraEnvs as a complete list.

If you redefine extraEnvs in your custom values file or through an overlay, you override the entire default set of environment variables shipped with the Observe Agent chart — not just the ones you specify.

That means:

Setting one custom environment variable without re-including the defaults will cause the rest (like OBSERVE_CLUSTER_NAME, K8S_NODE_NAME, etc.) to be lost.

:cross_mark: Example of a Problem

If you define only:

extraEnvs:
  - name: TOKEN
    valueFrom:
      secretKeyRef:
        name: external-secret-observe-token
        key: OBSERVE_TOKEN

Then all other default variables (like OBSERVE_CLUSTER_NAME, OBSERVE_CLUSTER_UID, etc.) are dropped.

Result:

  • The agent loses cluster identity context.
  • Metadata enrichment fails.
  • Metrics and logs may appear without proper tags in Observe.

:white_check_mark: Correct Way — Merge All Required Variables

Always include the full default environment variable set when customizing extraEnvs.

extraEnvs:
  - name: TOKEN
    valueFrom:
      secretKeyRef:
        name: external-secret-observe-token
        key: OBSERVE_TOKEN
        optional: false

  - name: OBSERVE_CLUSTER_NAME
    valueFrom:
      configMapKeyRef:
        name: cluster-name
        key: name

  - name: OBSERVE_CLUSTER_UID
    valueFrom:
      configMapKeyRef:
        name: cluster-info
        key: id

  - name: K8S_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

  - name: K8S_NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

If you need to add custom variables (for example, CUSTOM_ENV=production), append them to this list — never replace it.

:brain: Why GitOps and CD Tools Make This Easy to Miss

When using GitOps or declarative tools (like ArgoCD, Spinnaker, Flux, or Terraform Helm releases):

  • The tool applies your provided values file exactly as written.
  • It doesn’t “merge” nested lists such as extraEnvs; it replaces them entirely.
  • So a partial override will wipe out Helm’s defaults silently.

:white_check_mark: Best Practice Summary

Recommendation Description
:magnifying_glass_tilted_left: Verify all required env vars Check that all standard extraEnvs are present before deployment.
:puzzle_piece: Use complete list overrides When overriding Helm values, redefine all env vars together.
:brick: Validate post-deployment Run kubectl describe pod <agent-pod> → confirm environment variables are populated.
:locked: Keep credentials secure Reference tokens and IDs from Secrets and ConfigMaps using valueFrom.

:white_check_mark: In short:

If you redefine even one environment variable using extraEnvs, Helm will replace the entire block — not merge it.
Always include all required variables to avoid breaking cluster identification and telemetry flow.