> ## Documentation Index
> Fetch the complete documentation index at: https://docs.risolu.to/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Monitor Risoluto with Prometheus metrics, real-time SSE event streams, audit logs, X-Request-ID tracing, Sentry error tracking, and structured Pino logs.

Risoluto provides multiple observability surfaces: Prometheus metrics, real-time event streams, audit logs, request tracing, and structured logs.

<Tabs>
  <Tab title="Prometheus">
    ## Prometheus Metrics

    Available at `GET /metrics` in Prometheus exposition format. Scrape at 10-15 second intervals.

    ```bash theme={null}
    curl -s http://127.0.0.1:4000/metrics
    ```

    ### Metric reference

    <AccordionGroup>
      <Accordion title="HTTP metrics">
        | Metric                                   | Type      | Labels             | Description                  |
        | ---------------------------------------- | --------- | ------------------ | ---------------------------- |
        | `risoluto_http_requests_total`           | Counter   | `method`, `status` | Total HTTP requests          |
        | `risoluto_http_request_duration_seconds` | Histogram | `method`           | Request latency distribution |

        **Useful queries:**

        ```promql theme={null}
        # Error rate (5xx)
        sum(rate(risoluto_http_requests_total{status=~"5.."}[5m]))
        / sum(rate(risoluto_http_requests_total[5m]))

        # P95 latency
        histogram_quantile(0.95, rate(risoluto_http_request_duration_seconds_bucket[5m]))
        ```
      </Accordion>

      <Accordion title="Orchestrator metrics">
        | Metric                              | Type    | Labels   | Description                           |
        | ----------------------------------- | ------- | -------- | ------------------------------------- |
        | `risoluto_orchestrator_polls_total` | Counter | `status` | Poll cycles (success, error, skipped) |

        **Useful queries:**

        ```promql theme={null}
        # Polls per minute
        rate(risoluto_orchestrator_polls_total{status="success"}[5m]) * 60

        # Stall detection — no polls in 10 minutes
        increase(risoluto_orchestrator_polls_total[10m]) == 0
        ```
      </Accordion>

      <Accordion title="Agent metrics">
        | Metric                              | Type    | Labels      | Description                                         |
        | ----------------------------------- | ------- | ----------- | --------------------------------------------------- |
        | `risoluto_agent_runs_total`         | Counter | `outcome`   | Agent completions (completed, failed, oom, stalled) |
        | `risoluto_container_cpu_percent`    | Gauge   | `container` | Sandbox container CPU usage percentage              |
        | `risoluto_container_memory_percent` | Gauge   | `container` | Sandbox container memory usage percentage           |

        **Useful queries:**

        ```promql theme={null}
        # Success rate (1h)
        sum(rate(risoluto_agent_runs_total{outcome="completed"}[1h]))
        / sum(rate(risoluto_agent_runs_total[1h]))

        # Runs by outcome (stacked chart)
        sum by (outcome) (increase(risoluto_agent_runs_total[1h]))

        # Containers at CPU saturation
        max by (container) (risoluto_container_cpu_percent) > 90
        ```
      </Accordion>

      <Accordion title="Webhook metrics">
        Webhook delivery pipeline metrics — ingest volume, duplicate rejection, DLQ routing, backlog depth, and end-to-end processing latency.

        | Metric                                        | Type      | Labels                | Description                                                |
        | --------------------------------------------- | --------- | --------------------- | ---------------------------------------------------------- |
        | `risoluto_webhook_deliveries_total`           | Counter   | `provider`, `status`  | Raw webhook deliveries received                            |
        | `risoluto_webhook_duplicates_total`           | Counter   | `provider`            | Deliveries rejected as duplicates by the idempotency store |
        | `risoluto_webhook_events_processed_total`     | Counter   | `provider`, `outcome` | Events that passed validation and reached the processor    |
        | `risoluto_webhook_processor_retries_total`    | Counter   | `provider`, `reason`  | Retry attempts inside the webhook processor                |
        | `risoluto_webhook_dlq_total`                  | Counter   | `reason`              | Events moved to the dead-letter queue                      |
        | `risoluto_webhook_subscription_checks_total`  | Counter   | `provider`, `status`  | Subscription health checks performed                       |
        | `risoluto_webhook_backlog_count`              | Gauge     | `provider`            | Current count of unprocessed webhook events                |
        | `risoluto_webhook_dlq_count`                  | Gauge     | —                     | Current count of events in the dead-letter queue           |
        | `risoluto_webhook_last_delivery_age_seconds`  | Gauge     | `provider`            | Seconds since the last successful delivery                 |
        | `risoluto_webhook_processing_latency_seconds` | Histogram | `provider`            | End-to-end webhook processing latency                      |

        **Useful queries:**

        ```promql theme={null}
        # Webhook backlog growing
        max(risoluto_webhook_backlog_count) > 100

        # DLQ accumulating
        increase(risoluto_webhook_dlq_total[15m]) > 0

        # Delivery staleness (webhook feed broken)
        max(risoluto_webhook_last_delivery_age_seconds) > 900

        # P95 webhook processing latency
        histogram_quantile(0.95, rate(risoluto_webhook_processing_latency_seconds_bucket[5m]))
        ```
      </Accordion>
    </AccordionGroup>

    ### Scrape configuration

    <CodeGroup>
      ```yaml Docker network theme={null}
      scrape_configs:
        - job_name: risoluto
          static_configs:
            - targets: ["risoluto:4000"]
          metrics_path: /metrics
          scrape_interval: 10s
      ```

      ```yaml Host mode theme={null}
      scrape_configs:
        - job_name: risoluto
          static_configs:
            - targets: ["host.docker.internal:4000"]
          metrics_path: /metrics
          scrape_interval: 10s
      ```
    </CodeGroup>

    ### Alert rules

    ```yaml theme={null}
    groups:
      - name: risoluto
        rules:
          - alert: HighErrorRate
            expr: >
              rate(risoluto_http_requests_total{status=~"5.."}[5m])
              / rate(risoluto_http_requests_total[5m]) > 0.05
            for: 5m
            labels:
              severity: warning
            annotations:
              summary: "HTTP error rate above 5%"

          - alert: AgentRunFailures
            expr: rate(risoluto_agent_runs_total{outcome="failed"}[15m]) > 0
            for: 15m
            labels:
              severity: critical
            annotations:
              summary: "Agent runs failing consistently"

          - alert: PollStalled
            expr: increase(risoluto_orchestrator_polls_total[10m]) == 0
            for: 10m
            labels:
              severity: warning
            annotations:
              summary: "Orchestrator hasn't polled in 10 minutes"
    ```
  </Tab>

  <Tab title="Dashboard">
    ## Dashboard

    The built-in dashboard at `http://127.0.0.1:4000` provides real-time visibility without external tooling.

    | View                | What it shows                                            |
    | ------------------- | -------------------------------------------------------- |
    | **Board**           | Kanban board with issue cards grouped by state           |
    | **Overview**        | Aggregate stats: active agents, success rate, total cost |
    | **Issue inspector** | Per-issue timeline, attempt history, agent output        |
    | **Live feed**       | Streaming events as they happen (via SSE)                |

    ### Key API endpoints for dashboard data

    ```bash theme={null}
    # Full orchestrator state
    curl -s http://127.0.0.1:4000/api/v1/state | jq

    # Runtime metadata (version, data dir, provider)
    curl -s http://127.0.0.1:4000/api/v1/runtime | jq

    # Issue detail with recent events
    curl -s http://127.0.0.1:4000/api/v1/NIN-6 | jq

    # All attempts for an issue
    curl -s http://127.0.0.1:4000/api/v1/NIN-6/attempts | jq
    ```
  </Tab>

  <Tab title="API">
    ## API Observability Endpoints

    ### Event stream (SSE)

    Connect to `GET /api/v1/events` for real-time orchestrator events:

    ```bash theme={null}
    curl -N http://localhost:4000/api/v1/events
    ```

    Events are delivered as Server-Sent Events. Each event includes a `type` field and a JSON `data` payload. See the [Dashboard guide](/guides/dashboard#real-time-events) for the full event type reference.

    ### Audit log

    `GET /api/v1/audit` returns the audit trail for config and secret mutations:

    ```bash theme={null}
    curl -s http://127.0.0.1:4000/api/v1/audit | jq
    ```

    Each entry records the operation, path, timestamp, and source (API, wizard, CLI). The audit log is append-only and stored alongside the encrypted credential store.

    ### State and runtime

    ```bash theme={null}
    # Full orchestrator state — all tracked issues, queues, active agents
    curl -s http://127.0.0.1:4000/api/v1/state | jq

    # Runtime metadata — version, data dir, provider, uptime
    curl -s http://127.0.0.1:4000/api/v1/runtime | jq

    # State transitions log
    curl -s http://127.0.0.1:4000/api/v1/transitions | jq
    ```
  </Tab>
</Tabs>

## Request Tracing

Every request gets an `X-Request-ID` header:

* **Incoming**: If the client sends `X-Request-ID`, it is preserved
* **Generated**: Otherwise a UUID v4 is assigned
* **Response**: The ID is always returned in the response headers

Use this ID to correlate logs, metrics, and audit entries for a single request.

## Error Tracking

Sentry-compatible error tracking when `SENTRY_DSN` is set:

```bash theme={null}
export SENTRY_DSN=https://your-key@sentry.io/project-id
```

When enabled: exceptions are captured with full stack traces, breadcrumbs track the last 100 operations, and context (issue identifier, attempt count) is attached to every error. DSN is redacted in log output.

When `SENTRY_DSN` is not set, a no-op tracker is used with zero overhead.

## Process Logs

Logs are emitted to **stdout** via Pino in structured format:

| Variable              | Values                  | Default  |
| --------------------- | ----------------------- | -------- |
| `RISOLUTO_LOG_FORMAT` | `logfmt`, `json`        | `logfmt` |
| `LOG_LEVEL`           | `trace` through `fatal` | `info`   |

```bash theme={null}
# Persist logs to file
node dist/cli/index.js --port 4000 2>&1 | tee risoluto.log

# JSON format for log aggregators (Loki, Datadog, etc.)
RISOLUTO_LOG_FORMAT=json node dist/cli/index.js --port 4000
```

## Data Persistence

All attempt and event data is stored in SQLite (`risoluto.db`) with WAL mode:

```
.risoluto/
├── risoluto.db           # Attempts, events, issue index
├── risoluto.db-shm       # WAL shared-memory
├── risoluto.db-wal       # Write-ahead log
├── config/               # Operator config overlay
├── secrets.enc           # Encrypted credential store
├── secrets.audit.log     # Access audit trail
└── master.key            # Encryption master key
```

### Querying data

```bash theme={null}
# Via API
curl -s http://127.0.0.1:4000/api/v1/state | jq
curl -s http://127.0.0.1:4000/api/v1/NIN-6/attempts | jq

# Via CLI helper
./risoluto MT-42
./risoluto --attempt <attempt-id>

# Direct SQLite (read-only via WAL)
sqlite3 .risoluto/risoluto.db \
  "SELECT attempt_id, issue_identifier, status FROM attempts ORDER BY started_at DESC LIMIT 10;"
```

## What's Next

<CardGroup cols={2}>
  <Card title="Monitoring Stack" icon="chart-line" href="/recipes/monitoring-stack">
    Set up Prometheus and Grafana with pre-built queries and alerts.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/operating/troubleshooting">
    Diagnose common failures and recovery procedures.
  </Card>

  <Card title="Notifications" icon="bell" href="/guides/notifications">
    Configure Slack notifications for agent lifecycle events.
  </Card>

  <Card title="Dashboard Guide" icon="chart-mixed" href="/guides/dashboard">
    Board view, issue inspector, and live event stream.
  </Card>
</CardGroup>