Data Enrichment Using Reference Tables

UI - Dashboards & Worksheets
, ,
1

What are Reference Tables in Observe?

Do you ever find yourself staring at a cryptic GUID and wishing you had more context? Or need to add static data to your Observe workspace to enrich your understanding of machine data? If so, you’re likely looking for Reference Tables.

Reference tables are a powerful feature in Observe that let you integrate long-lived business context directly with your machine data. They are essentially a way to add static, supplementary data into Observe to give your core data more meaning. You can create these tables from CSV files via the UI, Observe API, or even using Terraform.

Where to Learn More

For a deeper dive into how to use and create these tables, you can find the official documentation here: docs.observeinc.com/en/latest/content/reference/ReferenceTables.html.

2

I love this feature, but we have too many customers with too much data per customer, so the 10 MB limit doesn’t work for us.
Currently we jam a full dump through the collector every 24 hours, and build a resource dataset, but this is more wasteful than what we need.
Can I create a virtual table by unioning more than one reference table?
Can I provide a larger reference table in a bucket URL somewhere instead?
Can I import a reference table from Snowflake?

3

Thanks for the question. In your use cases, do you have an idea what size might work for you? I can check into this size restriction and see if it might be possible to raise that if we have customers that are running into this like you are.

4

Hey @jwatte for your other questions:

  • Can I create a virtual table by unioning more than one reference table?

    • This should be possible by using join to combine data from multiple reference tables based on a common key. Here’s an example:
    join on (productId = @"Reference Tables/Products".id), productName:@"Reference Tables/Products".name, price:@"Reference Tables/Products".priceUsd

  • Can I provide a larger reference table in a bucket URL somewhere instead?

    • Not possible yet, we have this in the backlog

  • Can I import a reference table from Snowflake?

    • Not possible yet, in the backlog as well
5

@jwatte , although the default limit is 10 MB, if you submit a support ticket we can work with you to update the limit for your tenant.

6

This sounds good!

Each of our customers is over a kilobyte of data and when there’s millions of them we’re talking gigabytes. I’ll connect with you offline on specifics.