signoz-investigating-alerts

Files (3)

• SKILL.md
• baseline-comparison.md · references/
• neighbor-signals.md · references/

SKILL.md (first lines)

# Alert Investigate

Diagnose why a SigNoz alert fired: correlate the alert's own signal with
neighbor signals around the fire window and surface a ranked list of likely
causes with evidence. Companion to `signoz-explaining-alerts` (explain
decodes the rule statically; investigate diagnoses a specific incident).

## Prerequisites

Needs SigNoz MCP tools (`signoz_get_alert`, `signoz_get_alert_history`,
`signoz_execute_builder_query`, `signoz_query_metrics`,
`signoz_search_traces`, `signoz_search_logs`, `signoz_get_trace_details`,
etc.). Confirm `signoz_*` tools are available before starting; if not, run
`signoz-mcp-setup` to initialize/repair the MCP connection. Without the
server there is no way to ground the analysis.

## When to use

Use when the user wants to:
- Understand why a specific alert fired.
- Find the root cause of a recent incident triggered by an alert.
- Correlate the alert's signal with related metrics, traces, and logs.
- Distinguish "real signal" fires from flapping or threshold-mistuning.

Do NOT use when the user wants to:
- Understand what an alert monitors → `signoz-explaining-alerts`.
- Create a new alert → `signoz-creating-alerts`.
- Modify an alert (raise threshold, add hysteresis) → call
  `signoz_update_alert` directly.
- Run a free-form investigation with no alert anchor →
  `signoz-generating-queries`.

## Required inputs

| Input | Required | Source if missing |
|---|---|---|
| Alert identifier (rule ID or name) | yes | `$ARGUMENTS[0]` or recent context |
| Time window | no | default to most recent fire from `signoz_get_alert_history` |

Fuzzy alert name (best-effort, read-only):
1. Call `signoz_list_alert_rules`, paginate, fuzzy-match the name.
2. State the interpretation ("Investigating fire of 'High Error Rate -
   Checkout' (id 42) at 14:32 UTC. If you meant a different alert or fire,
   tell me.") and proceed.

If no firing transition exists in the queried lookback, **stop** — there is
nothing to investigate. Respond:
> "Alert '[name]' has not fired in the last 7d, so there is no fire window
> to investigate. Use `signoz-explaining-alerts` to walk through the rule,
> or check whether the alert is enabled."

## Workflow

Three tiers with strict early-stop gates. Tier 1 always runs; Tier 2 runs
only if Tier 1 confirms a real fire; Tier 3 runs only if Tier 2 surfaces
correlated anomalies. Skipping the gates fires hundreds of unnecessary
trace/log queries on quiet alerts.

### Step 1: Resolve alert + fire window (Tier 0)

1. Resolve the alert id via `signoz_list_alert_rules` (paginated) if not
   given.
2. Call `signoz_get_alert` for the full rule config (query, threshold,
   resource scope the alert evaluated).
3. Call `signoz_get_alert_history` with `timeRange: "7d"`, `order: "desc"`,
   and `state` omitted so the timeline includes firing and inactive
   transitions. Paginate only when `data.nextCursor` exists (completeness
   note reports `hasMore: true`): pass it as `cursor`, replace `timeRange`
   with the note's resolved absolute `start`/`end`, and preserve the same
   state/filter (including omission) and order. Stop when `nextCursor` is
   absent / `hasMore: false`; never use `offset` or page fullness. If a
   later intentional filter means "resolved"/"recovered", use `inactive`.
   Enum: `inactive|pending|recovering|firing|nodata|disabled`; `recovering`
   is a transient keep-firing state, not resolution. Pattern analysis needs
   the complete transition set. Rows are emitted per label-group
   `fingerprint`; do not interleave them. From the response:
   - **Build rule-wide incident windows** from distinct rows where
     `overallStateChanged: true`: `overallState: "firing"` opens an incident;
     the next `overallState: "inactive"` closes it. Deduplicate matching
     timestamps and sort by `unixMilli` ascending before pairing. Default to
     the most recent incident unless `$ARGUMENTS[1]` selects another.
   - **Partition affected series by `fingerprint`** and retain each row's
     labels. Use only `stateChanged: true` rows to decide when that group
     fired/resolved and which group scopes Tier 1–3 queries.
   - **Note the fire pattern** from rule-wide transitions or one named
     fingerprint:
     - `one-off` → single fire with a long quiet period before/after.
     - `sustained` → fires that stayed firing for ≥ 1 evaluation cycle.
     - `flapping` → ≥ 3 fires within a 1h window, alternating fire/resolve.
     - `recurring` → fires at regular intervals (cron-like, e.g. hourly).
   - Never infer flapping from different fingerprints. The pattern guides
     tiers 2/3.

### Step 2: Tier 1 — what fired and how hard (always runs)

Establishes the fire is real (vs. transient threshold tickle or flap) and
quantifies magnitude.

1. Re-run the alert's primary query over `[fire_start - 30m, fire_start +
   30m]` via `signoz_execute_builder_query` for the alert's stored builder,
   formula, PromQL, or ClickHouse envelope. Preserve positive bounds/order
   so Tier 1 reproduces the stored alert. If a formula input is below 10000,
   record truncation risk and compare at 10000 before ruling groups out. For
   omissions, use 10000 on formula-input `builder_query` leaves and 100 on
   standalone/formula results. Find leaves from every formula expression
   (including `disabled: true` formulas), following references through the
   dependency graph. This walk sets comparison bounds only; it does not prove
   deterministic formula-to-formula order. Use v5 `order`: `__result desc`
   for metrics/formulas or primary aggregation desc for logs/traces, never
   dashboard `orderBy`. Time-series top-N ranks over the whole window and may
   omit a short-lived local spike.
2. Compute:
   - **Peak value** during the fire window.
   - **Threshold breach magnitude**: `(peak - threshold) / threshold * 100`
     for "above" alerts, inverted for "below".
   - **Fire duration**: the rule's overall firing→inactive interval, or the
     selected fingerprint's interval for a group-scoped investigation. Say
     which.
   - **Pre-fire baseline**: average in the 30m before fire start.

From therapys/dotagents, MIT.

More from therapys/dotagents