Modelwatch

Install

Works with Claude Code, Codex, VS Code, Cursor, Windsurf, Claude Desktop. Read the repository before you give any tool access to your data.

npx agentmarket install connector/bch1212-modelwatch

About

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list, install config and code. Checked Nov 1, 2026.

Tools

5 read · 3 write · 1 delete

Secrets

1 (API keys, tokens or passwords)

Install scripts

None found

Runs as

Node.js package

Tools (9)

Read from the source code; the running server may differ. Descriptions are the server’s own words; hidden characters are shown as [U+…].

• create_endpoint

  Register an LLM endpoint to monitor. The workspace must already have a stored API key for the provider (use the dashboard to add one). Returns the new endpoint's id.

• create_spec

  Create a behavioral spec. The first run after creation sets the baseline output; subsequent scheduled runs are scored against that baseline across 5 axes (semantic, format, refusal, length, contains). Provide a semantic similarity threshold or use the API default.

• get_drift_events

  Fetch recent drift events across the workspace, newest first. Each event has spec_id, run_id, severity, drift_score, summary, and created_at. Use this for weekly review or to drive an automation.

• get_health

  Per-spec health across the workspace: spec_id, spec_name, endpoint_name, status (green/yellow/red), last_drift_score and last_run_at. This is not a plan or billing KPI endpoint.

• get_spec_history

  Get the run history for a single spec. Returns each run's status, drift score, semantic similarity, format match, and timestamp — useful for trending charts and reasoning about when behavior shifted.

• list_endpoints

  List the LLM endpoints currently monitored in this workspace. Returns id, name, provider, model, is_active, created_at for each.

• list_specs

  List behavioral specs in the workspace. A spec is a stored prompt + expectation that ModelWatch replays on a schedule and diffs against a baseline. Returns id, name, input_text, schedule, has_baseline, and the parent endpoint_id.

• reset_baseline

  Clear a spec's baseline. The next run will record a new baseline instead of being diffed against the old one. Use this after you've intentionally changed your prompt template, model version, or the behavior you expect — otherwise every future run will look like drift.

• run_spec

  Run a spec on demand and return the drift score immediately. Use this to (1) set the baseline manually right after create_spec, or (2) sanity-check a spec without waiting for the next scheduled run. Returns the run status, output, drift score, semantic similarity, format match, refusal flag, and timestamp.

Signals

Stars

0

Forks

0

Open issues

0

Last commit (this folder)

Oct 7, 2026 (25 days ago)

Last release

Oct 7, 2026 (25 days ago)

From GitHub (fetched Oct 24, 2026).

Details

Registry name

io.github.bch1212/modelwatch

Version

0.1.3

Auth

API key

Packages

npm modelwatch-mcp

License

License unknown

Repository

github.com/bch1212/modelwatch

Website

modelwatch.app

First seen

Oct 24, 2026

Last checked

Nov 1, 2026

Source

Official MCP Registrygithub-graphql