Base WordPress Robots.txt

This also assumes you install Google XML Sitemaps as part of your regular routine. If you’re working on SEO enhancements I’d highly recommend it. You  will want to edit the domain name, or you can completely strip out that part and let the plugin add it in for you.

User-agent: *
Disallow: /index.php
Disallow: /wp-
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /comments/
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /cgi-bin
Disallow: /search
Disallow: /feed
Disallow: /rss
Disallow: /comments/feed
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Allow: /
Sitemap: http://www.domain.com/sitemap.xml.gz

If at any time you had the global translator plugin installed you’ll want to add these to that list as well:

 Disallow: /ar/
 Disallow: /bg/
 Disallow: /cr/
 Disallow: /cs/
 Disallow: /da/
 Disallow: /de/
 Disallow: /el/
 Disallow: /en/
 Disallow: /es/
 Disallow: /fi/
 Disallow: /fr/
 Disallow: /hi/
 Disallow: /hr/
 Disallow: /it/
 Disallow: /ja/
 Disallow: /ko/
 Disallow: /nl/
 Disallow: /no/
 Disallow: /pl/
 Disallow: /pt/
 Disallow: /ro/
 Disallow: /ru/
 Disallow: /sv/
 Disallow: /sw/
 Disallow: /zh/
 Disallow: /zh-CN/
 Disallow: /zh-cn/
 Disallow: /zt/
 Disallow: /translate/

But wait, why would you do that? The quick answer is I agree with the mindset of these posts:

• 4 Major Reasons why I uninstalled the Global Translator plugin
• Global Translator Plug-in review – To install or not to install