Check exactly what search engines can crawl on any site, and build a clean, error-free robots.txt in minutes
https://yoursite.com/robots.txt, not in a subfolder.A clean robots.txt is one piece of technical SEO — run a complete audit to check speed, mobile-friendliness, meta tags, and more.
Run Free Website Audit
Your robots.txt file is the first thing search engine crawlers check before they touch anything else
on your site, and a single misplaced rule can quietly de-index whole sections of it. This tool has two
halves. The tester fetches the live robots.txt from any domain (or lets you paste one
in directly), parses every User-agent group, Allow/Disallow rule, crawl-delay, and Sitemap entry, and
then lets you check exactly how a specific crawler — Googlebot, Bingbot, or any custom bot — would
treat a given URL path, using the same "most specific group, longest rule, Allow-wins-ties" matching
logic search engines actually use. The generator lets you build a robots.txt from
scratch with presets for common situations (allow everything, block a staging environment, WordPress
defaults) or fully custom rules, with a live preview you can copy or download straight to your site's
root. Whichever you use, always upload the result to https://yoursite.com/robots.txt —
anywhere else and crawlers will ignore it entirely.