🔌 How to Use Observe’s API for OPAL Queries

General / Other
, , ,
1

Hi everyone! :waving_hand:

This thread is a quick reference on how to use Observe’s API to pull data directly from your Observe datasets using OPAL queries. It’s based on our developer documentation, with some practical examples you can adapt.

1. Authentication (Bearer Token)

First, you’ll need a bearer token to authenticate your API requests.
:warning: This is different from the API token you can generate in the Observe UI.

Follow the instructions here: Observe API Authentication Guide.

Once you have your bearer_token and your observe_tenant name, you’re ready to build API calls.

2. Finding Your Dataset ID

  • Go to your Observe account and open the dataset you want to query.

  • The dataset_id will be in the dataset’s URL.

  • Example:

    https://<tenant>.observeinc.com/dataset/41007104

    → Here, 41007104 is your dataset_id.

3. Example OPAL Query Payload

Let’s start with an example payload (straight from the developer docs):

{
  "query": {
    "outputStage": "myStage",
    "stages": [
      {
        "input": [
          {
            "inputName": "main",
            "datasetId": "41007104"
          },
          {
            "inputName": "rds",
            "datasetPath": "Workspace.aws/RDS Cluster"
          }
        ],
        "stageID": "myStage",
        "pipeline": "pick_col timestamp, log, namespace, containerName, rdsArn\nleftjoin on(rdsArn=@rds.arn), db:@rds.DBClusterIdentifier\n"
      }
    ]
  },
  "rowCount": "2"
}

4. Building the cURL Command

Now let’s wrap this into a POST request with cURL. Replace placeholders with your values:

curl -H "Authorization: Bearer ${observe_tenant} ${bearer_token}" \
     -H "Content-Type: application/json" \
     -H "Accept: application/x-ndjson" \
     --request POST "https://${observe_tenant}.observeinc.com/v1/meta/export/query?startTime=2025-01-14T01:21:00Z&endTime=2025-01-24T01:24:00Z" \
     -d @- << EOF
{
  "query": {
    "stages": [
      {
        "stageId": "mainStage",
        "input": [
          {
            "inputName": "main",
            "datasetId": "${dataset_id}"
          }
        ],
        "pipeline": "pick_col timestamp, log\n"
      }
    ]
  },
  "outputStage": "mainStage",
  "rowCount": "200"
}
EOF

Key Things to Know:

  • inputName, stageId, and outputStage → you can name these anything you like.

  • rowCount → controls how many results you want.

  • pipeline → contains your OPAL code (modify as needed).

  • Query URL params (startTime, endTime) → define the timeframe for your data pull.

5. Example Run

For example, to get the last 200 log lines from a dataset between Jan 14 and Jan 24, you’d use:

--request POST "https://mytenant.observeinc.com/v1/meta/export/query?startTime=2025-01-14T01:21:00Z&endTime=2025-01-24T01:24:00Z"

6. Next Steps

  • Experiment with different OPAL pipelines (statsby, filter, groupby) to tailor results.

  • Swap Accept: application/x-ndjson with text/csv if you prefer CSV output.

  • Consider scripting queries with the Observe CLI for repeatability.

:speech_balloon: Discussion Prompt:
Has anyone here integrated Observe’s API queries into automation pipelines (e.g., CI/CD, monitoring scripts, or Terraform workflows)? Would love to see some real-world examples!