Free Robots.txt Generator Robots.txt Generator User-agent Section All robots (*) Googlebot Bingbot Yahoo! Slurp DuckDuckBot Baiduspider YandexBot Sogou Spider Facebook Bot Custom: Allow (one per line): Disallow (one per line): Add User-agent Section Sitemap URL: Crawl-delay: Preview Download Robots.txt