A/B test agent skills and MCP changes with Caliper
Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.
Skill by agentskillexchange
Chalk Labs
Need help deploying A/B test agent skills and MCP changes with Caliper?
Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.
Install
Works with Claude Code, Claude apps, Codex, Cursor, Gemini CLI, GitHub Copilot, VS Code, OpenCode, Goose, Windsurf. Read the repository before you give any tool access to your data.
npx agentmarket install skill/agentskillexchange-a-b-test-agent-skills-and-mcp-changes-with-caliper
Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.
Get the weekly diff
The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.
About
Use cases: backend, testing, agent-building
Reviews
Write a reviewNo reviews yet. Installed it? Reviews come from verified installers only. How reviews work
Permissions
What this skill can do and what it asks for, from its SKILL.md and code. Checked Oct 25, 2026.
- Tools
- No tools found
- Secrets
- None asked for
- Install scripts
- None found
Network hosts it talks to 1
codeload.github.com· found in install command
Files it touches
~/.claude/skills/a-b-test-agent-skills-and-mcp-changes-with-caliper
What's inside
- Skill name
A/B test agent skills and MCP changes with Caliper- Description
- Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable.
- License
- MIT
- Instructions
- 25 lines in SKILL.md · view on GitHub
Agent Skills spec issues
- name: only lowercase letters, digits and single hyphens allowed
- name: "A/B test agent skills and MCP changes with Caliper" does not match folder "ab-test-agent-skills-mcp-changes-with-caliper"
Files (1)
- SKILL.md
SKILL.md (first lines)
# A/B test agent skills and MCP changes with Caliper Use Caliper to run real agent tasks with and without a skill, MCP server, or rule change so reliability and token cost are measurable. ## Prerequisites Python 3.10+, Caliper CLI, target agent runtime such as Claude Code, Codex, Pi, or Hermes, skills or MCP servers under test ## Installation Install or set up from the source-backed instructions: Install the CLI with `pipx install caliper-eval` for direct runs, or add the agent skill with `npx skills@latest add edonadei/caliper`; then create an `.eval.yaml`, run `caliper run ... --k 3`, run an ablated control with `--ablate`, and compare results with `caliper compare`. - Source: https://github.com/edonadei/caliper ## Documentation - https://github.com/edonadei/caliper ## Source - [Agent Skill Exchange](https://agentskillexchange.com/skills/ab-test-agent-skills-mcp-changes-with-caliper/)
From agentskillexchange/skills, MIT.
More from agentskillexchange/skills
- 0.654score
Aident Loadout
agentskillexchange
Give Claude Code, Codex, Cursor, ChatGPT and other MCP clients 1,000+ apps and 400+ expert-built Skills through one reusable Aident Loadout setup, with…
SkillAFreshness: A★ 58Claude Code +9 - 0.654score
AIHOT News
agentskillexchange
Retrieve Chinese AI news, recent topic search, event timelines and briefs from AIHOT's anonymous read-only Agent API.
SkillAFreshness: A★ 58Claude Code +9 - 0.651score
Agent Browser Operator
agentskillexchange
Interactive browser skill for logged-in flows, dynamic pages, and session-aware site operations.
SkillAFreshness: A★ 58Claude Code +9 - 0.649score
AIChat All-in-One LLM CLI with Shell Assistant and RAG
agentskillexchange
AIChat is a comprehensive LLM command-line tool written in Rust that combines chat-REPL, shell command generation, RAG, AI tools, and multi-provider support…
SkillAFreshness: A★ 58Claude Code +9 - 0.649score
Agent Media UGC Video
agentskillexchange
Use Agent Media when an agent needs to create captioned vertical UGC videos, narrated b-roll overlays, reusable characters, images, and voice through a hosted…
SkillAFreshness: A★ 58Claude Code +9 - 0.614score
Check Agent Counterparty Trust with Agent Guild
agentskillexchange
Checks evidence about an unfamiliar autonomous agent before delegation, verifies signed Agent Guild passports, and returns a bounded hire, caution, or avoid…
SkillBFreshness: B★ 58Claude Code +9 - 0.610score
Add Claude design-system and accessibility workflows with UX/UI Agent Skills
agentskillexchange
Use UX/UI Agent Skills when Claude should generate tokens, component specs, accessibility audits, and framework-specific UI code from a repeatable design…
SkillBFreshness: B★ 58Claude Code +9 - 0.567score
Aider AI Pair Programming CLI Agent
agentskillexchange
Aider is an open-source AI pair programming tool that runs in your terminal, letting you collaborate with LLMs to edit code across your entire codebase.
SkillCFreshness: C★ 58Claude Code +9 - 0.567score
AI Engine WordPress MCP Server and AI Automation
agentskillexchange
AI Engine is a WordPress plugin by Meow Apps that connects sites to OpenAI, Claude, Gemini, and other models while exposing WordPress actions through MCP and…
SkillCFreshness: C★ 58Claude Code +9 - 0.567score
ACF Extended for Advanced Custom Fields Workflows
agentskillexchange
An ASE skill built around ACF Extended, the WordPress enhancement suite for Advanced Custom Fields that adds field types, admin improvements, front-end forms…
SkillCFreshness: C★ 58Claude Code +9 - 0.564score
Add graph-backed memory and context retrieval to agent workflows
agentskillexchange
Use Cognee to ingest project knowledge into graph and vector memory so agents can retrieve durable context across sessions and workflows.
SkillCFreshness: C★ 58Claude Code +9 - 0.564score
ACF to Gutenberg Block Migrator
agentskillexchange
Converts Advanced Custom Fields field groups into native Gutenberg blocks using the ACF Block API v2 and @wordpress/scripts build pipeline.
SkillCFreshness: C★ 58Claude Code +9
Signals
- Stars
- 58
- Forks
- 96
- Open issues
- 4
- Last commit (this folder)
- Sep 21, 2026 (1 month ago)
From GitHub (fetched Oct 24, 2026).
Details
- License
- MIT
- Repository
- github.com/agentskillexchange/skills
- Website
- agentskillexchange.com
- First seen
- Oct 24, 2026
- Last checked
- Oct 24, 2026
- Source
- GitHubgithub-graphql
More from agentskillexchange
- 0.654score
Aident Loadout
agentskillexchange
Give Claude Code, Codex, Cursor, ChatGPT and other MCP clients 1,000+ apps and 400+ expert-built Skills through one reusable Aident Loadout setup, with…
SkillAFreshness: A★ 58Claude Code +9 - 0.654score
AIHOT News
agentskillexchange
Retrieve Chinese AI news, recent topic search, event timelines and briefs from AIHOT's anonymous read-only Agent API.
SkillAFreshness: A★ 58Claude Code +9 - 0.651score
Agent Browser Operator
agentskillexchange
Interactive browser skill for logged-in flows, dynamic pages, and session-aware site operations.
SkillAFreshness: A★ 58Claude Code +9 - 0.649score
AIChat All-in-One LLM CLI with Shell Assistant and RAG
agentskillexchange
AIChat is a comprehensive LLM command-line tool written in Rust that combines chat-REPL, shell command generation, RAG, AI tools, and multi-provider support…
SkillAFreshness: A★ 58Claude Code +9 - 0.649score
Agent Media UGC Video
agentskillexchange
Use Agent Media when an agent needs to create captioned vertical UGC videos, narrated b-roll overlays, reusable characters, images, and voice through a hosted…
SkillAFreshness: A★ 58Claude Code +9 - 0.614score
Check Agent Counterparty Trust with Agent Guild
agentskillexchange
Checks evidence about an unfamiliar autonomous agent before delegation, verifies signed Agent Guild passports, and returns a bounded hire, caution, or avoid…
SkillBFreshness: B★ 58Claude Code +9
Done for you
Rather not set it up yourself?
Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.
See how we build AI agents →Read the case studies
Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.
- launches shipped
- 50+
- launches shipped
- media placements
- 500+
- media placements
- founders on every account
- 2
- founders on every account
Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block
Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.
Get A/B test agent skills and MCP changes with Caliper running in your business
Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.
Cite this page
A/B test agent skills and MCP changes with Caliper. Agent Mart. https://agentmart.dev/skills/agentskillexchange-a-b-test-agent-skills-and-mcp-changes-with-caliper (last updated 2026-10-24)