Observability

Risoluto provides multiple observability surfaces: Prometheus metrics, real-time event streams, audit logs, request tracing, and structured logs.

Prometheus
Dashboard
API

Prometheus Metrics

Available at GET /metrics in Prometheus exposition format. Scrape at 10-15 second intervals.

curl -s http://127.0.0.1:4000/metrics

Metric reference

HTTP metrics

Metric	Type	Labels	Description
`risoluto_http_requests_total`	Counter	`method`, `status`	Total HTTP requests
`risoluto_http_request_duration_seconds`	Histogram	`method`	Request latency distribution

Useful queries:

# Error rate (5xx)
sum(rate(risoluto_http_requests_total{status=~"5.."}[5m]))
/ sum(rate(risoluto_http_requests_total[5m]))

# P95 latency
histogram_quantile(0.95, rate(risoluto_http_request_duration_seconds_bucket[5m]))

Orchestrator metrics

Metric	Type	Labels	Description
`risoluto_orchestrator_polls_total`	Counter	`status`	Poll cycles (success, error, skipped)

Useful queries:

# Polls per minute
rate(risoluto_orchestrator_polls_total{status="success"}[5m]) * 60

# Stall detection — no polls in 10 minutes
increase(risoluto_orchestrator_polls_total[10m]) == 0

Agent metrics

Metric	Type	Labels	Description
`risoluto_agent_runs_total`	Counter	`outcome`	Agent completions (completed, failed, oom, stalled)
`risoluto_container_cpu_percent`	Gauge	`container`	Sandbox container CPU usage percentage
`risoluto_container_memory_percent`	Gauge	`container`	Sandbox container memory usage percentage

Useful queries:

# Success rate (1h)
sum(rate(risoluto_agent_runs_total{outcome="completed"}[1h]))
/ sum(rate(risoluto_agent_runs_total[1h]))

# Runs by outcome (stacked chart)
sum by (outcome) (increase(risoluto_agent_runs_total[1h]))

# Containers at CPU saturation
max by (container) (risoluto_container_cpu_percent) > 90

Webhook metrics

Webhook delivery pipeline metrics — ingest volume, duplicate rejection, DLQ routing, backlog depth, and end-to-end processing latency.

Metric	Type	Labels	Description
`risoluto_webhook_deliveries_total`	Counter	`provider`, `status`	Raw webhook deliveries received
`risoluto_webhook_duplicates_total`	Counter	`provider`	Deliveries rejected as duplicates by the idempotency store
`risoluto_webhook_events_processed_total`	Counter	`provider`, `outcome`	Events that passed validation and reached the processor
`risoluto_webhook_processor_retries_total`	Counter	`provider`, `reason`	Retry attempts inside the webhook processor
`risoluto_webhook_dlq_total`	Counter	`reason`	Events moved to the dead-letter queue
`risoluto_webhook_subscription_checks_total`	Counter	`provider`, `status`	Subscription health checks performed
`risoluto_webhook_backlog_count`	Gauge	`provider`	Current count of unprocessed webhook events
`risoluto_webhook_dlq_count`	Gauge	—	Current count of events in the dead-letter queue
`risoluto_webhook_last_delivery_age_seconds`	Gauge	`provider`	Seconds since the last successful delivery
`risoluto_webhook_processing_latency_seconds`	Histogram	`provider`	End-to-end webhook processing latency

Useful queries:

# Webhook backlog growing
max(risoluto_webhook_backlog_count) > 100

# DLQ accumulating
increase(risoluto_webhook_dlq_total[15m]) > 0

# Delivery staleness (webhook feed broken)
max(risoluto_webhook_last_delivery_age_seconds) > 900

# P95 webhook processing latency
histogram_quantile(0.95, rate(risoluto_webhook_processing_latency_seconds_bucket[5m]))

Scrape configuration

scrape_configs:
  - job_name: risoluto
    static_configs:
      - targets: ["risoluto:4000"]
    metrics_path: /metrics
    scrape_interval: 10s

scrape_configs:
  - job_name: risoluto
    static_configs:
      - targets: ["host.docker.internal:4000"]
    metrics_path: /metrics
    scrape_interval: 10s

Alert rules

groups:
  - name: risoluto
    rules:
      - alert: HighErrorRate
        expr: >
          rate(risoluto_http_requests_total{status=~"5.."}[5m])
          / rate(risoluto_http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "HTTP error rate above 5%"

      - alert: AgentRunFailures
        expr: rate(risoluto_agent_runs_total{outcome="failed"}[15m]) > 0
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Agent runs failing consistently"

      - alert: PollStalled
        expr: increase(risoluto_orchestrator_polls_total[10m]) == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Orchestrator hasn't polled in 10 minutes"

Dashboard

The built-in dashboard at http://127.0.0.1:4000 provides real-time visibility without external tooling.

View	What it shows
Board	Kanban board with issue cards grouped by state
Overview	Aggregate stats: active agents, success rate, total cost
Issue inspector	Per-issue timeline, attempt history, agent output
Live feed	Streaming events as they happen (via SSE)

Key API endpoints for dashboard data

# Full orchestrator state
curl -s http://127.0.0.1:4000/api/v1/state | jq

# Runtime metadata (version, data dir, provider)
curl -s http://127.0.0.1:4000/api/v1/runtime | jq

# Issue detail with recent events
curl -s http://127.0.0.1:4000/api/v1/NIN-6 | jq

# All attempts for an issue
curl -s http://127.0.0.1:4000/api/v1/NIN-6/attempts | jq

API Observability Endpoints

Event stream (SSE)

Connect to GET /api/v1/events for real-time orchestrator events:

curl -N http://localhost:4000/api/v1/events

Events are delivered as Server-Sent Events. Each event includes a type field and a JSON data payload. See the Dashboard guide for the full event type reference.

Audit log

GET /api/v1/audit returns the audit trail for config and secret mutations:

curl -s http://127.0.0.1:4000/api/v1/audit | jq

Each entry records the operation, path, timestamp, and source (API, wizard, CLI). The audit log is append-only and stored alongside the encrypted credential store.

State and runtime

# Full orchestrator state — all tracked issues, queues, active agents
curl -s http://127.0.0.1:4000/api/v1/state | jq

# Runtime metadata — version, data dir, provider, uptime
curl -s http://127.0.0.1:4000/api/v1/runtime | jq

# State transitions log
curl -s http://127.0.0.1:4000/api/v1/transitions | jq

Request Tracing

Every request gets an X-Request-ID header:

Incoming: If the client sends X-Request-ID, it is preserved
Generated: Otherwise a UUID v4 is assigned
Response: The ID is always returned in the response headers

Use this ID to correlate logs, metrics, and audit entries for a single request.

Error Tracking

Sentry-compatible error tracking when SENTRY_DSN is set:

export SENTRY_DSN=https://your-key@sentry.io/project-id

When enabled: exceptions are captured with full stack traces, breadcrumbs track the last 100 operations, and context (issue identifier, attempt count) is attached to every error. DSN is redacted in log output. When SENTRY_DSN is not set, a no-op tracker is used with zero overhead.

Process Logs

Logs are emitted to stdout via Pino in structured format:

Variable	Values	Default
`RISOLUTO_LOG_FORMAT`	`logfmt`, `json`	`logfmt`
`LOG_LEVEL`	`trace` through `fatal`	`info`

# Persist logs to file
node dist/cli/index.js --port 4000 2>&1 | tee risoluto.log

# JSON format for log aggregators (Loki, Datadog, etc.)
RISOLUTO_LOG_FORMAT=json node dist/cli/index.js --port 4000

Data Persistence

All attempt and event data is stored in SQLite (risoluto.db) with WAL mode:

.risoluto/
├── risoluto.db           # Attempts, events, issue index
├── risoluto.db-shm       # WAL shared-memory
├── risoluto.db-wal       # Write-ahead log
├── config/               # Operator config overlay
├── secrets.enc           # Encrypted credential store
├── secrets.audit.log     # Access audit trail
└── master.key            # Encryption master key

Querying data

# Via API
curl -s http://127.0.0.1:4000/api/v1/state | jq
curl -s http://127.0.0.1:4000/api/v1/NIN-6/attempts | jq

# Via CLI helper
./risoluto MT-42
./risoluto --attempt <attempt-id>

# Direct SQLite (read-only via WAL)
sqlite3 .risoluto/risoluto.db \
  "SELECT attempt_id, issue_identifier, status FROM attempts ORDER BY started_at DESC LIMIT 10;"

What’s Next

Monitoring Stack

Set up Prometheus and Grafana with pre-built queries and alerts.

Troubleshooting

Diagnose common failures and recovery procedures.

Notifications

Configure Slack notifications for agent lifecycle events.

Dashboard Guide

Board view, issue inspector, and live event stream.

Get Started

Core Concepts

Guides

Recipes

Operating

Prometheus Metrics

Metric reference

Scrape configuration

Alert rules

Dashboard

Key API endpoints for dashboard data

API Observability Endpoints

Event stream (SSE)

Audit log

State and runtime

Request Tracing

Error Tracking

Process Logs

Data Persistence

Querying data

What’s Next

Monitoring Stack

Troubleshooting

Notifications

Dashboard Guide

​Prometheus Metrics

​Metric reference

​Scrape configuration

​Alert rules

​Dashboard

​Key API endpoints for dashboard data

​API Observability Endpoints

​Event stream (SSE)

​Audit log

​State and runtime

​Request Tracing

​Error Tracking

​Process Logs

​Data Persistence

​Querying data

​What’s Next

Monitoring Stack

Troubleshooting

Notifications

Dashboard Guide

Prometheus Metrics

Metric reference

Scrape configuration

Alert rules

Dashboard

Key API endpoints for dashboard data

API Observability Endpoints

Event stream (SSE)

Audit log

State and runtime

Request Tracing

Error Tracking

Process Logs

Data Persistence

Querying data

What’s Next