Robots Txt Generator API

POST https://api.findutils.com/api/tools/robots-txt-generator/execute

Return a robots.txt file built from user-agent rules (allow, disallow, crawl-delay), sitemap URLs and an optional Host directive, or from a preset: allow-all, block-all, wordpress, ecommerce.

Open the tool Try it

Request body

application/json

rules

array optional

Rule groups: objects with user_agent (e.g. "*", "Googlebot", "GPTBot"), allow (array of paths), disallow (array of paths), crawl_delay (seconds). Overrides preset.
preset

string optional

Starting rule set when rules is omitted. One of allow-all · block-all · wordpress · ecommerce. Default "allow-all".
sitemaps

array optional

Absolute sitemap URLs.
host

string optional

Optional Host directive value (Yandex), e.g. "https://example.com".
include_comments

boolean optional

Prepend the generator comment header with the date. Default true.

Example arguments

Verified

{
  "preset": "allow-all",
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ],
  "include_comments": false
}

curl -X POST https://api.findutils.com/api/tools/robots-txt-generator/execute \
  -H "Content-Type: application/json" \
  -d '{
    "preset": "allow-all",
    "sitemaps": [
      "https://example.com/sitemap.xml"
    ],
    "include_comments": false
  }'

const response = await fetch('https://api.findutils.com/api/tools/robots-txt-generator/execute', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({"preset":"allow-all","sitemaps":["https://example.com/sitemap.xml"],"include_comments":false}),
});
const data = await response.json();

import requests

response = requests.post(
    "https://api.findutils.com/api/tools/robots-txt-generator/execute",
    json={"preset":"allow-all","sitemaps":["https://example.com/sitemap.xml"],"include_comments":False},
)
data = response.json()

$ch = curl_init('https://api.findutils.com/api/tools/robots-txt-generator/execute');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
    CURLOPT_POSTFIELDS => '{"preset":"allow-all","sitemaps":["https://example.com/sitemap.xml"],"include_comments":false}',
    CURLOPT_RETURNTRANSFER => true,
]);
$data = json_decode(curl_exec($ch), true);

Also an MCP tool

Call robots_txt_generator from Claude, Cursor or any MCP client. Same arguments, same output.

https://mcp.findutils.com

MCP reference

Rate limit: 60 a minute, per IP
Auth: None
REST id: robots-txt-generator
MCP name: robots_txt_generator
Category: SEO
Machine-readable: openapi.json · markdown.md

Request body

Example arguments

More SEO tools

More to explore