Why would a monitor sometimes fail to trigger alerts

Monitors & Alerts
, ,
1

Description
The monitor may fail to trigger alert during certain time periods when built using an Opal that contains a filter like:

filter label(^Resource) = "some string"

and the Resource is defined using make_resource.

Affected Component/Service/Version

  • Observe Resources (make_resource)
  • Monitors relying on Resource labels or lookups

Caused by

  • make_resource expiry value set shorter than the data refresh cadence.
    When a dataset is published containing a make_resource it will usually have an expiry value that defaults to 1 day, unless the valid_for option is specified. For example:

    make_resource options(expiry:13h),
  Tags,
  Configuration,
  Name,
  ServiceSubType,
  ARN,
  primary_key(AccountID, Region, Service, ID)

    If the make_resource expiry is shorter than the data refresh interval, Resources can “fall out of existence” before the next update arrives. For example, if updates only arrive once per day, then with an expiry of 13h the Resource will be missing for ~11 hours each day.

    During those 11h windows:

    • Monitors will see no matching rows.
    • Dashboards, worksheets, or ad-hoc queries will also return nothing.
    • This creates the appearance that the Resource has stopped existing, even though it still exists historically.

  • Resource disappears temporarily when no new events are observed.

  • Monitor evaluation engine runs within a window where the Resource has expired → no rows returned.

Resolution

  • Ensure the expiry value (or valid_for) on make_resource is longer than the longest data refresh interval.
    • Example: If agents send updates once every 24h, set expiry to > 24h (e.g., expiry: 25h or valid_for: 2d).
  • This guarantees the Resource remains valid across ingestion gaps, ensuring monitors always have rows to evaluate.

References

  • Observe Docs: make_resource
  • Internal note: Monitor preview queries may still show results for 24h windows, even if the real-time evaluation engine sees nothing. This explains the apparent inconsistency.
1 Like