You’ve probably been wondering which would be the best robots.txt file to use with WordPress sites.
Well in this case here it is:
User-Agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /archives/ Disallow: /page/ Disallow: /comments/feed/ Disallow: /index.php Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /tag Disallow: /author Disallow: /trackback Disallow: /*trackback Disallow: /*trackback* Disallow: /*/trackback Disallow: /feed/ Disallow: /xmlrpc.php Disallow: /?p=*
One might as well consider that nowadays the bots have the capability to ignore the robots.txt file, but since Google’s best practices recommends the use of robots.txt on all sites this should be a good asset.
Additionally one can add a sitemap string if a sitemap had already been created so that crawlers follow the sitemap instead of doing recursive crawls on all pages:
Obviously, you must replace website.com in the URL above with your own.