Robots.txt & AI Crawler Checker

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Connector by robots-check.quietworklabs.workers.dev · Domain verified by the MCP Registry

IndexedLicense unknownRemote (HTTP)

Chalk Labs

Need help deploying Robots.txt & AI Crawler Checker?

Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.

Book a call

Install

Works with Claude Code, VS Code, Cursor, Windsurf. Read the repository before you give any tool access to your data.

npx agentmarket install connector/robots-check-quietworklabs-workers-dev-robots-check

Asks which of your tools to add it to (Claude Code, Codex, Cursor, VS Code, Windsurf, Gemini CLI, Claude Desktop) and shows the change first. See https://agentmart.dev/cli.

Remote endpoint

https://robots-check.quietworklabs.workers.dev/mcp Remote (HTTP)

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

About

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Permissions

What this server can do and what it asks for, from its tool list and install config. Checked Oct 25, 2026.

Tools: 3 read · 0 write · 0 deleteRead-only
Secrets: None asked for
Install scripts: Nothing to install (hosted)
Runs as: Hosted (remote endpoint)

Tools by what they do 3

From the running server's own tool list (tools/list) on Oct 25, 2026.

Read 3 · Look things up without changing anything.

ai_crawler_access
robots_check
sitemap_check

Network hosts it talks to 1

robots-check.quietworklabs.workers.dev · found in remote endpoint, install command

Tools (3)

As the server listed them on Oct 25, 2026. Descriptions are the server’s own words; hidden characters are shown as [U+…].

ai_crawler_accessWhich AI crawlers a site allows or blocksread-only
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
robots_checkMay this user-agent crawl this URL? (RFC 9309)read-only
Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.
sitemap_checkFind and validate an XML sitemapread-only
Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.

Signals

Last release: Oct 18, 2026 (13 days ago)

Details

Registry name: dev.workers.quietworklabs.robots-check/robots-check
Version: 1.0.0
Auth: Auth not stated
License: License unknown
Website: robots-check.quietworklabs.workers.dev/
First seen: Oct 24, 2026
Last checked: Oct 24, 2026
Source: Official MCP Registry

Rank

of 44,833 in Connectors

Score

0.562

Security 30%not yet
Popularity 25%0.30
Freshness 20%1.00
Rating 15%not yet
Docs 10%0.62

How ranking works →

Grades

Freshness

Active in the last 30 days. Last activity 13 days ago.

–

Security

Security scan: in progress. Grades from our dependency, secret, code and prompt-injection checks appear here once they launch. Until then, check the code before you install.

Indexed

Found automatically from a public source. Not yet claimed by its publisher.

How grades work →

Availability

100% uptime, last 30 days (1 check)

Last check: 6 days ago
Status: Up
Median response: 648 ms

Up
Up, auth required
Down
Not checked
No data

A daily MCP handshake with the remote endpoint. “Auth required” means it answered and wants a login or key. “Not checked” means the URL needs your own account or blocks automated checks.

Is this yours?

Claim it to add pricing, screenshots and docs, and to get a verified badge.

Claim listing Suggest a fix

Report a security problem

Done for you

Rather not set it up yourself?

Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.

See how we build AI agents →Read the case studies

Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.

launches shipped: 50+; launches shipped
media placements: 500+; media placements
founders on every account: 2; founders on every account

Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block

Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.

Get Robots.txt & AI Crawler Checker running in your business

Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.

Need help deploying Robots.txt & AI Crawler Checker?

Call Get help

Cite this page

Robots.txt & AI Crawler Checker MCP server. Agent Mart. https://agentmart.dev/connectors/robots-check-quietworklabs-workers-dev-robots-check (last updated 2026-10-24)