Removing duplicate and unnecessary content in Wordpress helps with SEO. This is a base robots.txt compilation of what I start with on Wordpress sites.
This also assumes you install Google XML Sitemaps as part of your regular routine. If you’re working on SEO enhancements I’d highly recommend it. You will want to edit the domain name, or you can completely strip out that part and let the plugin add it in for you.
User-agent: * Disallow: /index.php Disallow: /wp- Disallow: /feed/ Disallow: /comments/feed/ Disallow: /comments/ Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /cgi-bin Disallow: /search Disallow: /feed Disallow: /rss Disallow: /comments/feed Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /feed/$ Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ Allow: / Sitemap: http://www.domain.com/sitemap.xml.gz
If at any time you had the global translator plugin installed you’ll want to add these to that list as well:
Disallow: /ar/ Disallow: /bg/ Disallow: /cr/ Disallow: /cs/ Disallow: /da/ Disallow: /de/ Disallow: /el/ Disallow: /en/ Disallow: /es/ Disallow: /fi/ Disallow: /fr/ Disallow: /hi/ Disallow: /hr/ Disallow: /it/ Disallow: /ja/ Disallow: /ko/ Disallow: /nl/ Disallow: /no/ Disallow: /pl/ Disallow: /pt/ Disallow: /ro/ Disallow: /ru/ Disallow: /sv/ Disallow: /sw/ Disallow: /zh/ Disallow: /zh-CN/ Disallow: /zh-cn/ Disallow: /zt/ Disallow: /translate/
But wait, why would you do that? The quick answer is I agree with the mindset of these posts:
Respond: Leave A Comment | Trackback URL
Entrupeners, Subscribe for the lastest tools, tips, and tutorials.
Leave a Reply
You must be logged in to post a comment.