Skip to content

Monitoring Setup

This document provides guidance on setting up monitoring and alerting for the Open Resource Broker.

Overview

Monitoring is essential for maintaining the health and performance of your Host Factory deployment. This guide covers:

  • System monitoring setup
  • Application metrics collection
  • Alerting configuration
  • Dashboard creation
  • Log aggregation

System Monitoring

Health Checks

The plugin provides built-in health check endpoints:

# Check provider health
orb providers health

# Check system status
orb system status

Metrics Collection

The plugin exposes metrics in Prometheus format:

  • Request counts and latencies
  • Provider operation metrics
  • Storage operation metrics
  • Error rates and types

Log Monitoring

Configure log aggregation for:

  • Application logs
  • Provider operation logs
  • Error logs and stack traces
  • Audit logs

Background Request Reconciliation

A long-running ORB server (REST API or daemon) can refresh in-flight requests from their providers on a timer, so stored request and machine state follows the provider without a client polling each request. List views serve stored state that was refreshed recently instead of calling the provider again.

Configure it under request in the configuration file:

Key Default Meaning
reconcile_enabled false Set to true to run the loop. Off by default because it makes periodic provider API calls, one per in-flight request per pass. While off, list views always refresh from the provider.
reconcile_interval_seconds 60 Seconds between passes.
reconcile_max_concurrency 4 Most requests refreshed against providers at the same time.
reconcile_request_timeout_seconds 30 Time limit for refreshing one request.
list_cache_max_age_seconds 30 A request refreshed within this window is listed from stored state. 0 always refreshes.

Requests of a provider whose circuit breaker is open are skipped until the circuit closes, and provider failures seen by the loop count toward that circuit. A failed provider call does not refresh a request's last-checked time.

Every write the loop makes re-reads the request first, so a request cancelled while its provider call was running stays cancelled. With the SQL store a conflicting write is detected and re-applied once; the JSON store has no version check, so a change that lands in the few milliseconds between that read and the save can still be lost.

The loop runs only in a server with server.workers set to 1; with more workers it is not started (a message is logged at startup), because each worker would run its own loop against the same providers and storage. It starts with the server; on shutdown calls already in flight get a few seconds to finish before the pass is cancelled. Several single-worker replicas that share one SQL or DynamoDB store each run their own loop; the writes stay safe, but each replica calls the providers for the same requests, so enable the loop on one replica only. One-shot CLI commands do not run it. The list-view caching described above is per process.

Alerting

Set up alerts for:

  • High error rates
  • Provider failures
  • Storage issues
  • Performance degradation

Dashboard Examples

Create dashboards to monitor:

  • Request volume and success rates
  • Provider performance metrics
  • Resource utilization
  • Error trends

Troubleshooting

Common monitoring issues and solutions:

  • Metric collection failures
  • Alert configuration problems
  • Dashboard display issues

For detailed troubleshooting, see the Troubleshooting Guide.