Performance Tip: Use make_interval for Ephemeral Things in Observe

General / Other
, ,
1

Problem
If you’re modeling short-lived events—like API calls, user sessions, or financial transactions—using make_resource can slow down performance and use up query credits. That’s because make_resource is intended for long-lived entities (e.g., hosts, containers, customers).

Solution

Convert your dataset from a resource model to an interval model by replacing make_resource with make_interval. This approach is more efficient for ephemeral data.

Why This Matters

  • Resource datasets are meant for objects that persist and potentially change state over time (think: hosts, customers, containers).

  • Interval datasets are ideal for things that are fleeting and static in nature (think: sessions, API calls, individual requests).

How-to: Using make_interval

make_interval options(expiry: 10s), valid_from, valid_to

  • options(expiry: 10s) — Sets how long the interval is considered valid.

  • valid_from, valid_to — Define the start and end timestamps for each interval.

Note: Older Observe datasets might use make_session, which is functionally similar:

make_session options(expiry:10s), request_name:any_not_null(name), request_tags:any_not_null(tags), group_by(request_id)

Example Use Case

Suppose you’re tracking customer requests that last only a few minutes, each made up of a handful of events. Use make_interval to represent each request as a self-contained time slice—this significantly reduces query complexity and improves performance.

Summary

Scenario Recommended Function Why It Works Better
Long-lived, mutable entities make_resource Designed for items with persistent states
Short-lived, static events make_interval Optimized for ephemeral, one-shot data points

Let’s Discuss!

Have you tried switching to make_interval already? How did it affect:

  • Query performance?

  • Observability cost or credits?

  • Data modeling workflows?

Share your questions or battle-tested patterns—this can really help other community members model efficiently and avoid performance pitfalls!

Let me know if you’d like help refining this further or adding sample OPAL pipelines that implement make_interval in context!