Aug
25
2010

Base WordPress Robots.txt

Removing duplicate and unnecessary content in Wordpress helps with SEO. This is a base robots.txt compilation of what I start with on Wordpress sites.

Share
Email

This also assumes you install Google XML Sitemaps as part of your regular routine. If you’re working on SEO enhancements I’d highly recommend it. You  will want to edit the domain name, or you can completely strip out that part and let the plugin add it in for you.

User-agent: *
Disallow: /index.php
Disallow: /wp-
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /comments/
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /cgi-bin
Disallow: /search
Disallow: /feed
Disallow: /rss
Disallow: /comments/feed
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Allow: /
Sitemap: http://www.domain.com/sitemap.xml.gz

If at any time you had the global translator plugin installed you’ll want to add these to that list as well:

 Disallow: /ar/
 Disallow: /bg/
 Disallow: /cr/
 Disallow: /cs/
 Disallow: /da/
 Disallow: /de/
 Disallow: /el/
 Disallow: /en/
 Disallow: /es/
 Disallow: /fi/
 Disallow: /fr/
 Disallow: /hi/
 Disallow: /hr/
 Disallow: /it/
 Disallow: /ja/
 Disallow: /ko/
 Disallow: /nl/
 Disallow: /no/
 Disallow: /pl/
 Disallow: /pt/
 Disallow: /ro/
 Disallow: /ru/
 Disallow: /sv/
 Disallow: /sw/
 Disallow: /zh/
 Disallow: /zh-CN/
 Disallow: /zh-cn/
 Disallow: /zt/
 Disallow: /translate/

But wait, why would you do that? The quick answer is I agree with the mindset of these posts:

Respond: Leave A Comment | Trackback URL

Entrupeners, Subscribe for the lastest tools, tips, and tutorials.


Leave a Reply

Custom Theme by Rob Malon | Content & Design © 2010 - RobMalon.Com - Chicago, Illinois