skill-audit:audit

Files (1)

• SKILL.md

SKILL.md (first lines)

# /skill-audit:audit — Skill Activation Experiment

Run a complete skill activation audit in 4 phases. Present results after each phase and ask the user before proceeding to the next.

## Allowed Tools

Read, Write, Edit, Glob, Grep, Bash, Task, AskUserQuestion,
mcp__cclogviewer__list_sessions, mcp__cclogviewer__list_projects,
mcp__cclogviewer__search_logs, mcp__cclogviewer__get_session_timeline,
mcp__cclogviewer__get_tool_usage_stats, mcp__cclogviewer__get_session_stats

---

## Phase 1: Skill Inventory & Directive Audit

### 1.1 Discover skills

Use Glob to find all `.claude/skills/*/SKILL.md` files in the current project.

### 1.2 Parse each skill

For each SKILL.md, read the file and extract:
- `name` from frontmatter
- `description` from frontmatter
- `keywords` from frontmatter
- Full markdown body

### 1.3 Score directive compliance (0–3)

Score each skill's `description` field:

| Points | Criterion | Pattern Examples |
|--------|-----------|-----------------|
| +1 | Activation directive | "ALWAYS invoke", "MUST use", "ALWAYS use this skill" |
| +1 | Negative constraint | "Do not ... directly", "Never attempt to ... without", "Do not run ... directly" |
| +1 | Keyword triggers present | `keywords:` field has 5+ entries |

### 1.4 Present results

Display a table:

```
Phase 1: Skill Inventory & Directive Compliance

| # | Skill | Description (first 60 chars) | Dir. Score | Issues |
|---|-------|------------------------------|------------|--------|
| 1 | dockerfile-generator | Docker expert. ALWAYS invoke... | 3/3 | — |
| 2 | git-workflow | Git version control expert... | 2/3 | Missing keyword triggers |
```

For any skill scoring < 3, show a specific recommendation:
- Missing activation directive: "Add 'ALWAYS invoke this skill when...' to description"
- Missing negative constraint: "Add 'Do not ... directly — use this skill first' to description"
- Missing keyword triggers: "Add a `keywords:` field with 5+ relevant terms"

### 1.5 Ask to continue

Ask the user which skills to test:
- **All** — test every discovered skill
- **Select specific** — let user pick from the list
- **Fix first** — pause to fix directive issues before testing

---

## Phase 2: Query Mining

### 2.1 Mine queries from session logs

Use the cclogviewer MCP tools to find real user queries that should have triggered each skill:

1. Call `mcp__cclogviewer__list_projects` to find the current project
2. Call `mcp__cclogviewer__list_sessions` to get recent sessions (limit 50)
3. For each skill, call `mcp__cclogviewer__search_logs` with each keyword from the skill's `keywords` field
4. Cross-reference with `mcp__cclogviewer__get_tool_usage_stats` to find sessions where the Skill tool was NOT invoked (missed activations)
5. Extract the user's original query text from matched log entries

### 2.2 Generate contextual queries (if needed)

If fewer than 5 real queries found per skill:

1. Use Glob to find relevant project files (e.g., for a dockerfile skill, look for `Dockerfile*`, `docker-compose*`, `.dockerignore`)
2. Generate queries that reference actual project context:
   - "Write a Dockerfile for the Express app in src/server/"
   - "Help me resolve the merge conflict in package.json"
3. Ensure queries are specific enough to avoid clarification responses
4. Tag generated queries as `generated`

### 2.3 Present query list

For each skill, show the queries:

```
Phase 2: Query Mining

dockerfile-generator (5 queries):
  [log] "help me write a dockerfile for this project"
  [log] "containerize the api server"
  [gen] "Write a Dockerfile for the Express app in src/server/"
  [gen] "Create a multi-stage Docker build for production"
  [gen] "Add a docker-compose.yml for local development"

git-workflow (4 queries):
  [log] "resolve the merge conflict in main"
  [log] "squash my last 3 commits"
  [gen] "Help me rebase feature-auth onto main"
  [gen] "Undo the last commit without losing changes"
```

### 2.4 Ask to continue

Ask the user:
- **Run inline** — execute trials now in this session
- **Generate script** — write a `run-trials.sh` script for later execution
- **Edit queries** — let user modify the query list first

---

## Phase 3: Activation Experiment

From SeleznovIvan/claude-skills-test, MIT.

More from SeleznovIvan/claude-skills-test