Dealing with Wordpress and the robots.txt file

If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!

It’s funny to me that there seems to be so much foolishness around sitemaps and robots.txt files, especially when it comes to wordpress.

And this is vitally important to get this straight! If you are considering monetizing your wordpress site, you need to get your on-site SEO perfect!

Well, I’ve been using xml sitemap generator for some time and it works quite well with one exception. If you try to make the setting for robots.txt to display your sitemap location, well, it doesn’t work. Now, I don’t blame this on the sitemap plugin, but wordpress itself.

It seems that wordpress doesn’t create the dynamic robots.txt file. Maybe I have file permission issues, but I’m not sure.

And I want to be sure! And I want it to be automated! Why would I have to fool around and wonder if something as important as a robots.txt file be left to chance?

Now, the robots.txt file is important in that it is a file that is available to search engine robots as a starting point for reviewing and indexing your site’s information. If you have problems here, it could cause adverse issues. At worst you are not indexed properly. At best, you don’t rank as well.

Back to my story. Because I couldn’t properly depend on the robots.txt file to protect my site from getting same pages indexed as duplicate content and have it display the location of my sitemap.xml file, in an automated fashion, I’ve just skipped the sitemap.xml file listing.

After some reading, it seems that this is a fairly important component of onsite SEO. So, I had to find a better solution. Well, I believe that I have! Peter Coughlin on his website has created a magic plugin!

Take a cruise over to and check out his plugin, Robots.txt Wordpress plugin. Now, this is a simple but slick plugin. And does it work well!

Within 5 minutes of downloading it, I had it installed, activated, modified and running! Next, I check to see if the robots.txt file was been written. All I have to say is, two thumbs up!

There was one small change that I did make to the robots.txt file. Oh, didn’t I mentioned that you can actually modify your robots.txt within wordpress? Well now you can!

Below is the lines that I added near the bottom of the plugin configuration:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /tag
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/
Disallow: /*?

Now, I apologize, you’ll have to look and see what lines I’ve added. It took me about a minute to add the extra lines in the disallows.

My recommendation? Give it a try. Just make sure you’ve removed any manually made robots.txt file and then you should be golden!

So, Peter, 5 stars mate!