ai-bug-triage

Files (3)

• SKILL.md
• ci-failure-analysis.md · references/
• classification-taxonomy.md · references/

SKILL.md (first lines)

<objective>
A hybrid pipeline for bug classification, deduplication, and ticket generation. Deterministic fingerprinting handles deduplication (what LLMs are bad at); LLM handles explanation, severity assessment, and ticket writing (what LLMs are good at).

**Key reframe:** The LLM is best at explaining and routing, not deduplication. Teach agents to DESIGN the pipeline, not BE the pipeline.
</objective>

**Before starting:** Check for `.agents/qa-project-context.md` in the project root. It contains tech stack, component mapping, and known flaky areas that improve classification accuracy.

---

## Discovery Questions

Before building or using a triage pipeline, clarify:

1. **What is the failure source?**
   - CI pipeline logs (GitHub Actions, GitLab CI, Jenkins, CircleCI)
   - Test framework output (Playwright, Jest, pytest, Vitest)
   - Production error monitoring (Sentry, Datadog, Bugsnag)
   - Manual bug reports from QA or users

2. **What is the ticket destination?**
   - Jira, Linear, GitHub Issues, Azure DevOps, Shortcut
   - What fields are required? (component, severity, priority, labels)
   - What workflows exist? (triage board, auto-assignment rules)

3. **What is the deduplication scope?**
   - Same test run? Same sprint? Same release? All time?
   - Do you already have fingerprinting? What is the current duplicate rate?

4. **What approval workflow is needed?**
   - Auto-create tickets with human review?
   - Suggest tickets for human approval before creation?
   - Auto-close duplicates? (dangerous -- require approval)

5. **What historical data exists?**
   - Past bug reports with resolution data?
   - Flaky test history? Known environment issues?
   - Component ownership mapping?

---

## Core Principles

1. **Deterministic first, LLM second.** Use stable, reproducible fingerprinting for deduplication and clustering. Use LLM only for tasks requiring understanding: severity classification, root cause hypothesis, and human-readable ticket writing.

2. **Normalize before comparing.** Raw CI logs are full of timestamps, port numbers, process IDs, and random suffixes that make identical failures look different. Strip all noise before fingerprinting.

3. **Fingerprints are anchored to stable elements.** Exception type, top stack frames, test name, error message template, and URL pattern are stable. Timestamps, request IDs, and ephemeral ports are not.

4. **Human approval before destructive actions.** Auto-closing a ticket as duplicate or auto-merging reports requires human confirmation. False deduplication wastes more time than manual triage.

5. **Classification drives routing.** The value of triage is not the label itself but the routing decision it enables: which team, what priority, what SLA.

6. **Track triage accuracy.** Measure how often auto-classification matches human judgment. Below 85% accuracy, the pipeline needs tuning.

---

## The Pipeline

```
CI Log / Error Report
  │
  ▼
Step 1: NORMALIZE
  Strip timestamps, process IDs, ports, random suffixes, ANSI codes
  │
  ▼
Step 2: EXTRACT STABLE ANCHORS
  Exception type, top N stack frames, test name, error message template, URL pattern
  │
  ▼
Step 3: HASH CANONICAL FORM
  Deterministic fingerprint from ordered anchors
  │
  ▼
Step 4: CLUSTER NEAR-DUPLICATES
  Similarity scoring for non-identical but related failures
  │
  ▼
Step 5: LLM CLASSIFY
  Severity, component, suspected root cause, failure category
  │
  ▼
Step 6: LLM GENERATE TICKET
  Title, description, repro steps, evidence, suggested assignee
  │
  ▼
Step 7: HUMAN APPROVAL
  Review before create/close/merge
```

### Step 1: Normalize

Strip noise that makes identical failures look different.

**Normalization rules (apply in order):**

```
1. Strip ANSI color codes:        \x1b\[[0-9;]*m → ""
2. Strip timestamps:              \d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}[.\d]*Z? → "<TIMESTAMP>"
3. Strip UUIDs:                   [0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} → "<UUID>"
4. Strip process IDs:             pid[=: ]\d+ → "pid=<PID>"
5. Strip port numbers:            :\d{4,5}(?=[\s/]) → ":<PORT>"
6. Strip temp file paths:         /tmp/[^\s]+ → "<TMPPATH>"
7. Strip memory addresses:        0x[0-9a-f]{8,16} → "<ADDR>"
8. Strip random suffixes:         [-_][a-z0-9]{6,8}(?=\.) → "<RAND>"
9. Strip request IDs:             (?:request[_-]?id|trace[_-]?id|correlation[_-]?id)[=: ]["']?[a-zA-Z0-9-]+ → "<REQ_ID>"
10. Collapse whitespace:          \s+ → " "
```

**Example:**

```
Before: 2025-03-22T14:32:01.456Z [pid=42891] Error: Connection refused at 127.0.0.1:54321
        request_id=abc-123-def-456
After:  <TIMESTAMP> [pid=<PID>] Error: Connection refused at 127.0.0.1:<PORT>
        <REQ_ID>
```

From JaiminVadadoriya/Distributed-Cloud-Storage---Sync-Platform, MIT.

More from JaiminVadadoriya/Distributed-Cloud-Storage---Sync-Platform