Where should the robots.txt file be hosted?

The robots.txt file must always be placed at the absolute root of your domain (e.g. https://example.com/robots.txt).

How can I block AI scrapers like GPTBot or CCBot?

Add specific User-agent blocks: User-agent: GPTBot followed by Disallow: / to prevent OpenAI crawling while allowing search engine indexing.

Does Disallow in robots.txt remove pages from Google index?

No. Disallow prevents Googlebot from crawling page content, but URLs can still be indexed if linked externally. Use a 'noindex' meta tag to remove pages from search results.

Can I declare multiple sitemaps in robots.txt?

Yes. Multiple Sitemap: lines can be declared at the bottom of the file to guide search crawlers to your XML sitemaps.

Robots.txt Generator

Related Tools

Explore Crawler Optimization Suite

XML Sitemap Generator

Build standards-compliant sitemap.xml files focused on URLs and accurate lastmod dates, with image entries and sitemap index output. changefreq/priority are optional legacy metadata Google ignores.

Indexing & SitemapsOpen →

Meta Tag & Social Preview Generator

Generate standard HTML meta tags and Next.js App Router metadata with character count safety checks.

SEOOpen →

URL Parser & Query String Inspector

Break down URLs into protocol, hostname, port, pathname, hash, and interactive query parameters.

URL & WebOpen →

Word & Character Counter

Count words, characters, sentences, paragraphs, reading time, and speaking time in real time.

Text AnalysisOpen →

Inputs stay in browser memory. No external request is required for this operation, and zero data leaves your device.