Free SEO tools · Technical

Robots.txt generator

Choose which folders to keep crawlers out of, add your sitemap, and decide how AI crawlers are treated. Copy the result to /robots.txt at your site's root.

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://yourwebsite.com/sitemap.xml

Questions

Does robots.txt hide pages from Google?No. It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of results.
Should I block AI crawlers?Blocking AI search bots keeps you out of ChatGPT, Claude and Perplexity answers. Many businesses allow search bots and block only training bots.

Related tools

XML sitemap generator

Crawl a site (up to 100 pages) and get a sitemap.xml.

Redirect and status checker

Follow a URL and see every redirect hop and status code.

Indexability checker

Can Google index this page? Status, canonical, noindex and robots.txt.

Heading structure checker

See a page's H1–H3 outline and fix missing or skipped headings.

Image alt text checker

List every image on a page and whether it has alt text.

Page speed checker

Mobile and desktop speed scores with Core Web Vitals.

Hreflang tag generator

Language and region tags for multilingual or multi-country sites.

Social preview checker

See how a link looks when shared, and which social tags are missing.