Skip to content

ADR-0019: Observability Visualization Backend — SigNoz (Grafana Stack Descoped)

  • Status: ✅ Accepted
  • Date: 2026-07-17 (SigNoz accepted 2026-06; Grafana Stack descoped 2026-07-17)
  • Owners: @ilhan, @team/platform
  • Context: The runner/agent observability project (PL-105, "Collection and Visualization of Agent & Runner Data via OpenTelemetry") needed a visualization backend for the cloud-mac slice: macOS host machines and ephemeral tart runner VMs report metrics, traces, and logs through a vendor-neutral OTEL Collector gateway. Signal production and transport are identical for any backend — only the gateway exporter config and the deployed backend stack differ. The stack is deliberately separate from the existing self-hosted Grafana+Loki. A backend had to be picked for the first deliverable (2026-06-30) while keeping the door open for others.

Decision

SigNoz (Option B) is the visualization backend. It runs on a single OVH/Proxmox VM (docker compose: SigNoz + ClickHouse behind nginx; TLS at the edge) and is live with the project's dashboards (fleet inventory, cloud-mac health, build analytics, pipeline self-health) and the alert rule set.

The Grafana Stack (Option A) is DESCOPED as of 2026-07-17 — no Grafana deploy, dashboards, alerts, or parallel-profile maintenance remain in project scope. The gateway stays vendor-neutral: exporter profiles are split into signoz.yaml (active) and grafana.yaml (prepared, inactive) in the appcircle-observability repo, so adopting Grafana or any OTLP-compatible backend later is an exporter-config swap that requires no agent changes — it is simply no longer a deliverable.

Options Considered

  • Option A — Grafana Stack (Prometheus + Tempo + Loki + Grafana)
    • Pros: most flexible dashboards/alerts; team already operates Grafana for self-hosted; deepest ecosystem (Alertmanager, recording rules, Thanos).
    • Cons: four components to operate; highest setup effort; dashboards/alerts authored for SigNoz would need re-authoring (no 1:1 portability).
  • Option B — SigNoz (SigNoz + ClickHouse) — selected
    • Pros: metrics + traces + logs in one UI; OTEL-native (gateway exports OTLP straight to it); lowest setup effort (two services via docker compose); fast path for the June slice.
    • Cons: dashboards/alerts less flexible than Grafana; scaling at very high volumes harder (acceptable at current fleet size).
  • Option C — Existing enterprise tooling (Datadog / Elastic / Dynatrace via OTLP)
    • Only relevant when a customer already runs one; handled by adding a gateway exporter, agents untouched. Not applicable to our own cloud fleet.

Consequences

  • Positive:
    • June deliverable de-risked: one OTLP endpoint, two backend services; live since the 2026-07-10 Runner Observability demo.
    • Descoping Grafana removes the dual-authoring burden — dashboards/alerts can use SigNoz-native features freely (v5 dashboard schema, multi-threshold alert rules, embedded-alertmanager email channels).
    • Telemetry conventions are backend-independent; signals produced now remain valid if a different backend is ever adopted.
  • Negative:
    • Dashboards and alert rules are SigNoz-specific; a future backend change means re-authoring them (accepted, recorded here).
    • Very-high-volume scaling may eventually require ClickHouse tuning or a backend revisit.
  • Follow-ups / migrations:
    • Production rollout of the stack (PL-326) and retention/VM sizing (PL-188).
    • Customer-facing documentation (PL-327).
  • Related issues/PRs: Linear PL-109 (ADR issue) · PL-105 project · PL-114 (deploy/dashboards/alerts) · PL-326 (prod rollout) · PL-188 (retention/sizing)
  • Source ADR in the project repo: appcircle-observability/docs/adr-0001-backend-selection.md (Accepted 2026-06, amended 2026-07-17 with the descope)
  • Supersedes / superseded by: —