Best robots.txt file for WordPress sites

Home » Content Management Systems » Best robots.txt file for WordPress sites
August 10, 2017 Content Management Systems, Wordpress No Comments

You’ve probably been wondering which would be the best robots.txt file to use with WordPress sites.

Well in this case here it is:

User-Agent: *

Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /archives/
Disallow: /page/
Disallow: /comments/feed/
Disallow: /index.php
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /tag
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /feed/
Disallow: /xmlrpc.php
Disallow: /?p=*

One might as well consider that nowadays the bots have the capability to ignore the robots.txt file, but since Google’s best practices recommends the use of robots.txt on all sites this should be a good asset.

Additionally one can add a sitemap string if a sitemap had already been created so that crawlers follow the sitemap instead of doing recursive crawls on all pages:

Sitemap: https://website.com/sitemap.xml

Obviously, you must replace website.com in the URL above with your own.

LEAVE A COMMENT