Read a site's robots.txt and sitemap, find mistakes, and test whether a crawler may fetch a given page.
This tool needs JavaScript. Please enable it and reload.