How do I create Google Cloud Platform (GCP) Monitoring Pollers Using Terraform?

Data Ingestion
, , ,
1

For GCP deployments, you may want to use terraform for the GCP Resource Collection side as well as the GCP Metric collection side.

image
image1540Ă—1316 190 KB

Setting up Observe API Token and GCP Metrics Poller with Terraform

This guide walks you through creating an Observe API token and using Terraform to set up a Google Cloud Platform (GCP) metrics poller for Observe.

Prerequisites

  • Access to an Observe instance with admin privileges
  • A GCP project with appropriate permissions
  • Terraform installed (version 1.x)
  • gcloud CLI tool (optional, for service account setup)

Part 1: Creating an Observe API Token

Option 1: Create API Token via Observe UI

  1. Access the Authtokens UI:

    • Log into your Observe instance
    • Navigate to Account Settings > Authtokens
    • This UI is available to all admin users

  2. Create a new API token:

    • Click the Create API Token button
    • Configure the following settings:
      • User: Select the user to bind the token to
      • Name: Provide a friendly name (e.g., “Terraform GCP Poller”)
      • Description: Optional description of the token’s purpose
      • Expiration: Set expiration date or “Never expires”
      • Extension: Configure token validity extension after each use

  3. Save the token:

    • Copy the generated token immediately
    • Store it securely (you cannot view it again after closing the dialog)

Option 2: Create API Token via cURL (Local Auth)

For users with username/password authentication:

curl -H 'Content-Type: application/json' \
  https://YOUR_CUSTOMER_ID.observeinc.com/v1/login \
  -d '{"user_email":"your-email@example.com", "user_password":"your-password", "tokenName":"GCP Terraform Token"}'

Replace:

  • YOUR_CUSTOMER_ID with your 12-digit Observe customer ID
  • your-email@example.com with your Observe login email
  • your-password with your Observe password

The response will contain an access_key that you can use for API authentication.

Option 3: Create API Token via cURL (SSO)

For SSO-enabled instances, follow the three-step process:

  1. Initiate token request:
curl -d '{"userEmail":"your-email@example.com", "integration":"observe-tool-terraform","clientToken":"terraform setup"}' \
  https://YOUR_CUSTOMER_ID.observeinc.com/v1/login/delegated

  1. Approve the request:

    • Open the URL from the response in your browser
    • Log in via SSO and approve the request

  2. Retrieve your token:

curl -X GET https://YOUR_CUSTOMER_ID.observeinc.com/v1/login/delegated/SERVER_TOKEN

Use the serverToken from step 1 in place of SERVER_TOKEN.

Part 2: Setting up GCP Service Account

Required GCP Roles

Your GCP service account needs the following roles for metrics polling:

  • Monitoring Viewer (roles/monitoring.viewer)
  • Cloud Asset Viewer (roles/cloudasset.viewer)
  • Browser (roles/browser)

Create Service Account via GCP Console

  1. Navigate to IAM & Admin:

    • Go to the GCP Console
    • Select your project
    • Navigate to IAM & Admin > Service Accounts

  2. Create Service Account:

    • Click Create Service Account
    • Provide a name (e.g., “observe-metrics-poller”)
    • Add description: “Service account for Observe GCP metrics polling”

  3. Grant Required Roles:

    • Add the following roles to the service account:
      • Monitoring Viewer
      • Cloud Asset Viewer
      • Browser

  4. Generate Service Account Key:

    • Click on the created service account
    • Go to the Keys tab
    • Click Add Key > Create new key
    • Select JSON format
    • Download and securely store the JSON key file

Create Service Account via gcloud CLI

# Set your project ID
export PROJECT_ID="your-gcp-project-id"

# Create service account
gcloud iam service-accounts create observe-metrics-poller \
  --description="Service account for Observe GCP metrics polling" \
  --display-name="Observe Metrics Poller"

# Grant required roles
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:observe-metrics-poller@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/monitoring.viewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:observe-metrics-poller@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/cloudasset.viewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:observe-metrics-poller@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/browser"

# Create and download service account key
gcloud iam service-accounts keys create observe-sa-key.json \
  --iam-account=observe-metrics-poller@$PROJECT_ID.iam.gserviceaccount.com

Part 3: Terraform Configuration

Directory Structure

Create the following directory structure:

gcp-metrics-poller/
├── main.tf
├── variables.tf
├── providers.tf
├── terraform.tfvars
└── observe-sa-key.json

providers.tf

terraform {
  required_providers {
    observe = {
      source  = "observeinc/observe"
      version = "~> 0.14"
    }
  }
}

# Configure the Observe provider
provider "observe" {
  customer  = var.observe_customer_id
  api_token = var.observe_api_token
  domain    = var.observe_domain  # Optional: defaults to observeinc.com
}

variables.tf

variable "observe_customer_id" {
  description = "Observe customer ID (12-digit number)"
  type        = string
}

variable "observe_api_token" {
  description = "Observe API token for authentication"
  type        = string
  sensitive   = true
}

variable "observe_domain" {
  description = "Observe domain (e.g., observeinc.com, observe-staging.com)"
  type        = string
  default     = "observeinc.com"
}

variable "gcp_project_id" {
  description = "GCP project ID to monitor"
  type        = string
}

variable "gcp_service_account_key_file" {
  description = "Path to GCP service account JSON key file"
  type        = string
  default     = "observe-sa-key.json"
}

variable "poller_name" {
  description = "Name for the GCP metrics poller"
  type        = string
  default     = "gcp-metrics-poller"
}

variable "polling_interval" {
  description = "How frequently to poll for metrics (minimum: 1m0s)"
  type        = string
  default     = "5m0s"
}

variable "workspace_name" {
  description = "Observe workspace name"
  type        = string
  default     = "Default"
}

variable "datastream_name" {
  description = "Observe datastream name for GCP data"
  type        = string
  default     = "GCP"
}

variable "include_metric_prefixes" {
  description = "List of GCP metric type prefixes to include"
  type        = list(string)
  default = [
    "monitoring.googleapis.com",
    "pubsub.googleapis.com",
    "storage.googleapis.com",
    "compute.googleapis.com",
    "cloudsql.googleapis.com",
    "container.googleapis.com",
    "run.googleapis.com",
    "bigquery.googleapis.com",
    "cloudfunctions.googleapis.com",
    "logging.googleapis.com",
    "iam.googleapis.com"
  ]
}

variable "exclude_metric_prefixes" {
  description = "List of GCP metric type prefixes to exclude"
  type        = list(string)
  default     = ["aws.googleapis.com"]
}

main.tf

# Read the GCP service account key file
locals {
  gcp_service_account_key = jsondecode(file(var.gcp_service_account_key_file))
}

# Look up existing workspace
data "observe_workspace" "default" {
  name = var.workspace_name
}

# Look up or create datastream for GCP data
data "observe_datastream" "gcp" {
  workspace = data.observe_workspace.default.oid
  name      = var.datastream_name
}

# Create the GCP metrics poller
resource "observe_poller" "gcp_metrics" {
  workspace  = data.observe_workspace.default.oid
  name       = var.poller_name
  datastream = data.observe_datastream.gcp.oid
  interval   = var.polling_interval

  gcp_monitoring {
    project_id = var.gcp_project_id
    json_key   = jsonencode(local.gcp_service_account_key)

    include_metric_type_prefixes = var.include_metric_prefixes
    exclude_metric_type_prefixes = var.exclude_metric_prefixes
  }
}

# Output the poller information
output "poller_id" {
  description = "ID of the created GCP metrics poller"
  value       = observe_poller.gcp_metrics.id
}

output "poller_name" {
  description = "Name of the created GCP metrics poller"
  value       = observe_poller.gcp_metrics.name
}

terraform.tfvars (Template)

Create a terraform.tfvars file with your specific values:

# Observe Configuration
observe_customer_id = "123456789012"  # Your 12-digit Observe customer ID
observe_api_token   = "your-api-token-here"
observe_domain      = "observeinc.com"  # or your custom domain

# GCP Configuration
gcp_project_id = "your-gcp-project-id"
gcp_service_account_key_file = "observe-sa-key.json"

# Poller Configuration
poller_name       = "my-gcp-metrics-poller"
polling_interval  = "5m0s"
workspace_name    = "Default"
datastream_name   = "GCP"

# Customize metric prefixes if needed
include_metric_prefixes = [
  "monitoring.googleapis.com",
  "pubsub.googleapis.com",
  "storage.googleapis.com",
  "compute.googleapis.com"
]

Part 4: Deployment

Initialize and Deploy

  1. Initialize Terraform:
cd gcp-metrics-poller
terraform init
  1. Review the plan:
terraform plan
  1. Apply the configuration:
terraform apply
  1. Confirm deployment:
    • Type yes when prompted
    • Note the output values for the poller ID and name

Verify Deployment

  1. Check in Observe UI:

    • Navigate to your Observe instance
    • Go to Datastreams > GCP
    • Verify your GCP metrics poller is listed and active

  2. Monitor data ingestion:

    • Check the specified datastream for incoming GCP metrics data
    • Data should start appearing within the polling interval (default: 5 minutes)

Part 5: Customization and Troubleshooting

Customizing Metric Collection

You can customize which GCP metrics are collected by modifying the include_metric_prefixes and exclude_metric_prefixes variables:

# Include only specific services
include_metric_prefixes = [
  "compute.googleapis.com",      # Compute Engine
  "container.googleapis.com",    # GKE
  "cloudsql.googleapis.com",     # Cloud SQL
  "storage.googleapis.com"       # Cloud Storage
]

# Exclude AWS metrics that might appear in GCP monitoring
exclude_metric_prefixes = [
  "aws.googleapis.com"
]

Common Issues and Solutions

  1. Authentication Errors:

    • Verify your Observe API token is correct and not expired
    • Ensure the token has appropriate permissions

  2. GCP Permission Errors:

    • Confirm the service account has all required roles
    • Verify the service account key file is valid and accessible

  3. No Data Appearing:

    • Check that the GCP project has metrics to collect
    • Verify the datastream exists in the specified workspace
    • Ensure the polling interval allows sufficient time for data collection

  4. Terraform State Issues:

    • Use terraform refresh to sync state with actual resources
    • Consider using remote state storage for team environments

Monitoring and Maintenance

  • Token Rotation: Regularly rotate your Observe API tokens according to your security policies
  • Service Account Key Rotation: Rotate GCP service account keys periodically
  • Cost Monitoring: Monitor GCP API usage costs, especially for high-frequency polling
  • Performance Tuning: Adjust polling intervals based on your monitoring needs and API quotas

Security Best Practices

  1. Store Secrets Securely:

    • Never commit API tokens or service account keys to version control
    • Use environment variables or secret management systems
    • Consider using Terraform Cloud or similar for secure variable storage

  2. Principle of Least Privilege:

    • Grant only the minimum required GCP roles to the service account
    • Regularly audit and review permissions

  3. Network Security:

    • Ensure network connectivity between your Terraform execution environment and both Observe and GCP APIs
    • Consider using private endpoints where available

This completes the setup of your Observe GCP metrics poller using Terraform. The poller will now continuously collect metrics from your specified GCP project and send them to your Observe datastream for analysis and monitoring.

2 Likes