Bot Crawl Checker MCP tool
Network tool MCP
findutils:bot_crawl_checker Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or blocked for a URL by the site's robots.txt, with the matching rule and user-agent group per bot.
Arguments
application/json-
url
string required
Public http(s) URL to check, e.g. "https://example.com/blog/post".
Example arguments
Verified{
"url": "https://findutils.com/"
}More SEO tools
- Domain Rating Checker Return the Ahrefs Domain Rating (0-100) for up to 25 domains in one call, with a per-domain status (provider_unavailable means Ahrefs is…
- Faq Schema Generator Generate FAQPage JSON-LD structured data from question-answer pairs, and return the JSON-LD string, a ready <script> tag, the schema…
- Form To Json Schema Returns a form schema object ({ formId, method, action, fields, submitText }) extracted from HTML form markup.
- Geo Analyzer Return a Generative Engine Optimization (GEO) report for a public web page: an overall 0-100 score, per-category scores (technical…
- Graphql Playground Check a GraphQL query for balanced braces and, when a schema is supplied, unknown fields on Query/Mutation/Subscription types.
- Graphql Schema Validator Validate a GraphQL SDL schema and return errors, naming-convention warnings, and counts of types, queries, mutations, and subscriptions.