How do I add a filter to drop events using the Kubernetes Observe Agent?

Data Ingestion
,
1

This article guides you through updating the Observe Kubernetes Agent configuration to drop specific log entries using filters in the YAML configuration. The process involves modifying the Agent’s processor configuration to exclude unwanted logs based on specific patterns, ensuring cleaner data collection in your Kubernetes environment.

Steps to Update the Filter Configuration

Follow these steps to add a filter processor in the Observe Kubernetes Agent YAML configuration.

1. Retrieve the Current values.yaml File

If you already have your values.yaml file, you can edit it directly and skip to step 2. Otherwise, download the current configuration (replace <YOUR_NAMESPACE> with your actual namespace):

export NAMESPACE=<YOUR_NAMESPACE>
helm get values observe-agent -n $NAMESPACE -o yaml > current-values.yaml

2. Create and Edit the Updated values.yaml File

Make a copy of the current configuration and clean it up:

cp current-values.yaml updated-values.yaml
sed -i.bak '/^USER-SUPPLIED VALUES:$/d' updated-values.yaml

Open updated-values.yaml in a text editor and locate the agent.config.global section:

agent:
  config:
    global:

Add or update the agent.config.global.nodeLogsMetrics section to include a filter processor, this example uses filter/drop_failed_scrape:

agent:
  config:
    global:
      ...
      <keep existing values>
      ...
      nodeLogsMetrics:
        processors:
          filter/drop_failed_scrape:
            logs:
              exclude:
                match_type: regexp
                record_attributes:
                  - key: body
                    value: ".*Failed to scrape Prometheus endpoint.*"
                  - key: body
                    value: ".*failed to detect resource.*"
                  - key: body
                    value: ".*is being deprecated in favor.*"
          resourcedetection/cloud:
            detectors: [gcp]
        service:
          pipelines:
            logs:
              # add filter/drop_failed_scrape to the list of processors:
              processors: [filter/drop_failed_scrape, memory_limiter, k8sattributes, batch, resource/observe_common, attributes/debug_source_pod_logs]
            metrics/hostmetrics:
              processors: [memory_limiter, k8sattributes, batch, resource/observe_common, attributes/debug_source_hostmetrics, attributes/metrics, resource/metrics]
            metrics/kubeletstats:
              processors: [memory_limiter, k8sattributes, batch, resource/observe_common, attributes/debug_source_kubeletstats_metrics, attributes/metrics, resource/metrics]

The filter/drop_failed_scrape processor, in the example, excludes log entries matching specific patterns in the body attribute, such as “Failed to scrape Prometheus endpoint,” “failed to detect resource,” or “is being deprecated in favor.” This reduces noise in the observability data by dropping irrelevant or redundant logs.

The filter is configured under the agent.config.global.nodeLogsMetrics.processors section and must be included in the logs pipeline to take effect.

Customizing the Filter:

  • Update the record_attributes section under filter/drop_failed_scrape.logs.exclude to include additional patterns. For example, to drop logs containing “connection refused,” add:

    - key: body
  value: ".*connection refused.*"

  • Modify the match_type to strict if you need exact matches instead of regular expressions (regexp).

  • Ensure the filter/drop_failed_scrape processor is listed in the logs pipeline under service.pipelines.logs.processors.

4. Apply the Updated Configuration

Apply the updated configuration using Helm. Replace <YOUR_NAMESPACE> with your actual namespace:

helm upgrade observe-agent observe/agent -n $NAMESPACE -f updated-values.yaml

This command updates the Observe Agent with the new filter configuration.

5. Verify the Changes

  • Check the Observe dashboard to ensure the excluded logs no longer appear in the data pipeline.

  • If errors occur, inspect the agent logs for configuration issues or invalid regular expressions.