improve-skills

Files (2)

• SKILL.md
• evals-template.md · references/

SKILL.md (first lines)

# Improve Skills — Self-Healing Skill Optimizer

This skill does two things:

1. **SETUP** — Walk through creating an eval suite for a target skill. The evals are saved permanently inside that skill's own `references/evals.md` so they're always there when you need them.
2. **IMPROVE** — Load those stored evals and run an iterative improvement loop, updating the skill in place when something better is found.

---

## Detect the Mode

Choose the mode based on what the user said and what already exists.

**Use SETUP when:**
- The user says "set up evals for [skill]", "I want to improve [skill] someday", or similar
- The target skill has no `references/evals.md` yet

**Use IMPROVE when:**
- The user says "improve [skill]", "fix [skill]", "run evals for [skill]", or simply "improve-skills"
- The target skill already has `references/evals.md`

If the mode is unclear, ask: "Do you want to set up evals for the first time, or run the improvement loop against existing evals?"

---

## SETUP Mode — One-Time Eval Walkthrough

Guide the user through five questions in sequence. Do not move to the next question until the current one is confirmed.

### Question 1: Target Skill

Ask which skill they want to set up evals for, and confirm the exact skill name.

Load the skill using `load_skill` so you can read it before continuing. Summarize what the skill does in two or three sentences so the user can confirm you understand it correctly.

### Question 2: Data Source

Explain that instead of fixed test cases, the skill will pull real items from a SharePoint library or list every time it runs. This keeps tests grounded in current data and lets the skill adapt as the content evolves.

Ask:

- **Which library or list** on this site contains the content this skill processes? (For example: "Meeting Recordings", "Proposals Library", "Posts".)
- **How should items be sampled?** Options: newest items (default), random selection, or the user hand-picks specific items each session.
- **How many items per session?** Default is three. Maximum is five. More than five overwhelms the conversation and slows the loop without adding meaningful signal.

If the user wants to provide sample text directly instead of pointing at a library, accept that — note it in the data source field as "user-provided samples" and remind them they will need to paste content at the start of each IMPROVE session.

**If the library is empty when IMPROVE runs later**, the skill must stop immediately, tell the user the library has no content, and ask them to either add real items or paste sample text to continue.

### Question 3: Eval Criteria

Explain that you need three to six binary yes-or-no checks that define what a good output looks like for this skill on this kind of content. Before writing evals, read two or three items from the data source so the evals are grounded in what the content actually looks like — not a hypothetical version of it.

Share these rules for writing good evals:

- **Binary only.** Every eval must be yes or no — no scales, no partial credit, no "mostly passes."
- **Specific enough to be consistent.** "Is the output good?" is too vague. "Does the output include a direct recommendation in the first paragraph?" is testable.
- **Not so narrow the skill games it.** Avoid checks that only measure length or surface formatting, because the skill will optimize for those at the expense of everything else.
- **Three to six is the sweet spot.** More than six and the skill starts parroting the eval criteria rather than actually improving.

For each eval, capture:
- **Name** — a short label used in the results table
- **Question** — the yes-or-no check, phrased as a question
- **Pass** — a specific description of what "yes" looks like
- **Fail** — what triggers "no"

Work collaboratively with the user to write these. Propose drafts based on your reading of the skill, then refine together.

### Question 4: Configuration

Ask three things:

- How many items should be sampled per session? (Default: 3. Max: 5.)
- How many times should the skill be run per mutation cycle? (Default: 3. More runs reduce noise but take longer.)
- Is there a cap on total experiment cycles? (Default: none — the loop runs until the score ceiling is hit or the user stops it.)

### Question 5: Confirm and Save

Present the complete eval suite in the format shown in `references/evals-template.md`. Read that file now so you use the correct format.

Ask the user to confirm or make corrections. Then save the eval suite by calling `create_skill` with the target skill name and the file path `references/evals.md`.

Tell the user: "Evals saved. Anytime you want to improve this skill, just say 'improve [skill-name]' and I'll load these evals and run the loop automatically."

---

## IMPROVE Mode — Self-Healing Loop

### Step 1: Load Everything

Load two things before proceeding:

1. The current version of the target skill using `load_skill`
2. Its eval suite using `load_skill` with the file path `references/evals.md`

Parse the eval suite to extract: the data source configuration, the eval criteria, the configuration (items per session, runs per experiment, budget cap), and the improvement history table.

If `references/evals.md` does not exist, tell the user and switch to SETUP mode.

### Step 2: Sample Live Data

Pull fresh items from the data source every session — do not reuse items from a previous run.

Read items from the library or list specified in `evals.md` using the configured sampling method (newest, random, or user-picks). Pull the configured number of items (default: three, max: five).

**If the library is empty:** Stop immediately. Tell the user the library has no content and ask them to either add items or paste sample text directly. Do not proceed until you have real input to work with.

**If the user provides sample text instead of a library:** Accept the pasted content as this session's inputs. Note that these are user-provided samples, not live library data.

Present the sampled items to the user with a brief summary of each (title and a one-sentence description of the content). Ask: "Are these representative of what this skill typically handles? If not, tell me what's missing and I'll adjust." Wait for confirmation before continuing.

### Step 3: Check Eval Relevance

Before running a single experiment, look at the sampled items and ask whether the stored evals still reflect what good output looks like for *this* content.

Check for:

- **Evals that no longer apply.** If the data has changed in a way that makes a stored eval untestable or irrelevant, flag it. Example: an eval checking for deadline extraction is meaningless if today's transcripts contain no dates.
- **Patterns the evals miss.** If you notice something consistently present in the live data that no eval captures — a structural element, a type of content, a failure mode visible just from reading the inputs — name it. Ask the user whether to add an eval for it.

From zrosenfield/sharepoint-ai-skills, MIT.

More from zrosenfield/sharepoint-ai-skills