agent-evaluation
Use when evaluating agent outputs, designing benchmarks, or building self-evaluation rubrics.
Skill by HarleyBartles
Chalk Labs
Need help deploying agent-evaluation?
Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.
Install
Works with Claude Code, Claude apps, Codex, Cursor, Gemini CLI, GitHub Copilot, VS Code, OpenCode, Goose, Windsurf. Read the repository before you give any tool access to your data.
npx agentmarket install skill/harleybartles-agent-evaluation
Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.
Get the weekly diff
The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.
About
Use cases: agent-building
Reviews
Write a reviewNo reviews yet. Installed it? Reviews come from verified installers only. How reviews work
Permissions
What this skill can do and what it asks for, from its SKILL.md. Checked Oct 25, 2026.
- Tools
- No tools found
- Secrets
- None asked for
- Install scripts
- None found
Network hosts it talks to 1
codeload.github.com· found in install command
Files it touches
~/.claude/skills/agent-evaluation
What's inside
- Skill name
agent-evaluation- Description
- Use when evaluating agent outputs, designing benchmarks, or building self-evaluation rubrics.
- License
- MIT
- Metadata
- owner: Harley Bartles · status: active · source-id: agent-evaluation · source-path: skills/agent-evaluation/SKILL.md · provenance-name: Agent Evaluation first-party skill · source-category: first_party
- Instructions
- 27 lines in SKILL.md · view on GitHub
Files (6)
- SKILL.md
- openai.yaml · agents/
- CITATIONS.md · assets/authority/
- authority.yaml · assets/authority/
- source-map.yaml · assets/authority/
- operational-guidance.md · references/
SKILL.md (first lines)
# Agent Evaluation Use this skill when evaluating agent outputs, designing benchmarks, or building self-evaluation rubrics. ## When to Use - Evaluating an agent's output against a rubric. - Designing or running a benchmark. - Building a self-evaluation workflow. ## Core Pattern 1. Define the evaluation dimension (correctness, completeness, safety, concision) and scoring anchors before scoring. 2. Build or select benchmark tasks that reflect real deployment conditions and hold out a final test set. 3. Score outputs against rubrics or reference solutions, not just model self-ratings. 4. Report per-task and per-dimension results with failure modes and confidence intervals. 5. Keep the evaluation harness separate from the evaluated agent to avoid leakage. ## Common Mistakes - Reporting a single aggregate score without a breakdown. → Report per-dimension and per-task results. - Reusing the test set as a development target. → Hold out a final evaluation set. - Evaluating on politeness or verbosity instead of outcomes. → Score end results against criteria. Load `references/operational-guidance.md` for deeper coverage of benchmark design, scoring, and rubrics.
From HarleyBartles/agent-asset-marketplace, MIT.
More from HarleyBartles/agent-asset-marketplace
- 0.570score
agentic-harness
HarleyBartles
Use when designing, reviewing, or debugging agent loops, harness construction, tool/action spaces, or multi-agent orchestration.
SkillAFreshness: AClaude Code +9 - 0.570score
repo-standards
HarleyBartles
Use when aligning repository operating-model standards, assessing adoption, or deciding which focused standard owns a requested change.
SkillAFreshness: AClaude Code +9 - 0.570score
python
HarleyBartles
Use when writing, reviewing, or debugging Python code and the task calls for idiomatic language patterns, concurrency, testing, or type-safety guidance.
SkillAFreshness: AClaude Code +9 - 0.570score
api-design
HarleyBartles
Use when designing, reviewing, or versioning an HTTP API contract, choosing resource naming, or mapping OpenAPI structures to implementation boundaries.
SkillAFreshness: AClaude Code +9 - 0.569score
tracked-repo-hooks
HarleyBartles
Use when changing or assessing a repository's tracked pre-commit hook and hosted-CI parity under its adopted standard.
SkillAFreshness: AClaude Code +9 - 0.565score
semver
HarleyBartles
Use when choosing a product version, assessing compatibility or release identity, or tracing conflicting versions across package metadata, runtime, manifests…
SkillAFreshness: AClaude Code +9 - 0.565score
completed-artifact-custody
HarleyBartles
Use when a repository adopts completed-artifact-custody, or explicitly asks for its lifecycle, to assess plans, specifications, roadmaps, or checkpoints.
SkillAFreshness: AClaude Code +9 - 0.565score
gitflow
HarleyBartles
Use when choosing or assessing feature, release, or hotfix branch routes, release promotion, or integration reconciliation in a repository that adopts Gitflow.
SkillAFreshness: AClaude Code +9 - 0.565score
unslop
HarleyBartles
Use when recording recurring agent mistakes, maintaining repository slop guards, or assessing unslop profile effectiveness.
SkillAFreshness: AClaude Code +9 - 0.564score
runbook-composition
HarleyBartles
Use when adopting, authoring, routing, assessing, or updating repository lifecycle-stage runbooks.
SkillAFreshness: AClaude Code +9 - 0.564score
contribution-entrypoint
HarleyBartles
Use when creating, assessing, or maintaining a repository's root CONTRIBUTING.md contributor guidance.
SkillAFreshness: AClaude Code +9 - 0.564score
agent-doctrine-contracts
HarleyBartles
Use when adopting or assessing agent-facing doctrine and contracts, their repository locations, quality, or routes.
SkillAFreshness: AClaude Code +9
Signals
- Stars
- 0
- Forks
- 0
- Open issues
- 0
- Last commit (this folder)
- Sep 28, 2026 (1 month ago)
From GitHub (fetched Oct 24, 2026).
Details
- License
- MIT
- First seen
- Oct 24, 2026
- Last checked
- Oct 24, 2026
- Source
- GitHubgithub-graphql
More from HarleyBartles
- 0.570score
agentic-harness
HarleyBartles
Use when designing, reviewing, or debugging agent loops, harness construction, tool/action spaces, or multi-agent orchestration.
SkillAFreshness: AClaude Code +9 - 0.570score
api-design
HarleyBartles
Use when designing, reviewing, or versioning an HTTP API contract, choosing resource naming, or mapping OpenAPI structures to implementation boundaries.
SkillAFreshness: AClaude Code +9 - 0.570score
repo-standards
HarleyBartles
Use when aligning repository operating-model standards, assessing adoption, or deciding which focused standard owns a requested change.
SkillAFreshness: AClaude Code +9 - 0.570score
python
HarleyBartles
Use when writing, reviewing, or debugging Python code and the task calls for idiomatic language patterns, concurrency, testing, or type-safety guidance.
SkillAFreshness: AClaude Code +9 - 0.569score
tracked-repo-hooks
HarleyBartles
Use when changing or assessing a repository's tracked pre-commit hook and hosted-CI parity under its adopted standard.
SkillAFreshness: AClaude Code +9 - 0.565score
gitflow
HarleyBartles
Use when choosing or assessing feature, release, or hotfix branch routes, release promotion, or integration reconciliation in a repository that adopts Gitflow.
SkillAFreshness: AClaude Code +9
Done for you
Rather not set it up yourself?
Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.
See how we build AI agents →Read the case studies
Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.
- launches shipped
- 50+
- launches shipped
- media placements
- 500+
- media placements
- founders on every account
- 2
- founders on every account
Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block
Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.
Get agent-evaluation running in your business
Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.
Cite this page
agent-evaluation skill. Agent Mart. https://agentmart.dev/skills/harleybartles-agent-evaluation (last updated 2026-10-24)