Bot Crawl Checker API

Network tool

POST https://api.findutils.com/api/tools/bot-crawl-checker/execute

Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or blocked for a URL by the site's robots.txt, with the matching rule and user-agent group per bot.

Open the tool Try it

Request body

application/json

url

string required

Public http(s) URL to check, e.g. "https://example.com/blog/post".

Example arguments

Verified

{
  "url": "https://findutils.com/"
}

curl -X POST https://api.findutils.com/api/tools/bot-crawl-checker/execute \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://findutils.com/"
  }'

const response = await fetch('https://api.findutils.com/api/tools/bot-crawl-checker/execute', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({"url":"https://findutils.com/"}),
});
const data = await response.json();

import requests

response = requests.post(
    "https://api.findutils.com/api/tools/bot-crawl-checker/execute",
    json={"url":"https://findutils.com/"},
)
data = response.json()

$ch = curl_init('https://api.findutils.com/api/tools/bot-crawl-checker/execute');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
    CURLOPT_POSTFIELDS => '{"url":"https://findutils.com/"}',
    CURLOPT_RETURNTRANSFER => true,
]);
$data = json_decode(curl_exec($ch), true);

Also an MCP tool

Call bot_crawl_checker from Claude, Cursor or any MCP client. Same arguments, same output.

https://mcp.findutils.com

MCP reference

Rate limit: 60 a minute, per IP
Auth: None
REST id: bot-crawl-checker
MCP name: bot_crawl_checker
Category: SEO
Machine-readable: openapi.json · markdown.md

This tool fetches a fixed, hard-coded public upstream, never a private host. Results depend on that upstream.

Request body

Example arguments

More SEO tools

More to explore