Comments on: Dealing with Wordpress and the robots.txt file http://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/ Learn How to Use Wordpress For Fame and Fortune! Mon, 19 Jul 2010 04:07:15 -0700 http://wordpress.org/?v=2.9.2 hourly 1 By: Webmaster supporthttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-516 Webmaster support Wed, 28 Apr 2010 02:15:37 +0000 http://www.usingwp.com/?p=61#comment-516 Thanks for reply frank . Thanks for reply frank .

]]> By: fthomashttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-514 fthomas Tue, 27 Apr 2010 23:02:49 +0000 http://www.usingwp.com/?p=61#comment-514 Hi Rajnish,I took a look at your site and everything seems in order. Once Google indexes a page, the best you can do to have it removed is to send a request to google to remove those pages. But I would recommend just putting forwards to pages that are not in your site, that are indexed and take advantage of the free traffic.Frank Hi Rajnish,

I took a look at your site and everything seems in order. Once Google indexes a page, the best you can do to have it removed is to send a request to google to remove those pages. But I would recommend just putting forwards to pages that are not in your site, that are indexed and take advantage of the free traffic.

Frank

]]> By: Webmaster supporthttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-513 Webmaster support Tue, 27 Apr 2010 02:10:38 +0000 http://www.usingwp.com/?p=61#comment-513 Hi all my website is htttp://www. rajneesh .me/ and its has *removed* such kind of many urls indexed on google how can i stop them from indexing .I am also using robots.txtPlease help thanks in advance. Hi all my website is htttp://www. rajneesh .me/ and its has *removed* such kind of many urls indexed on google how can i stop them from indexing .I am also using robots.txt

Please help thanks in advance.

]]> By: retrohttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-389 retro Sun, 07 Mar 2010 08:59:20 +0000 http://www.usingwp.com/?p=61#comment-389 thanks for the help, this is really confusing, i removed my real robots.txt file and the virtual one shows up when try to check it out in the regular domain/robots.txt format, i guess all i can do now is find a plugin that will allow me to edit that virtual file. thanks for the help, this is really confusing, i removed my real robots.txt file and the virtual one shows up when try to check it out in the regular domain/robots.txt format, i guess all i can do now is find a plugin that will allow me to edit that virtual file.

]]> By: fthomashttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-384 fthomas Fri, 05 Mar 2010 23:06:50 +0000 http://www.usingwp.com/?p=61#comment-384 The robots.txt file and the sitemap.xml work hand in hand. I do believe that you need the robots.txt in the main directory but I'd double check that. Just be sure that the sitemap.xml that is called in the robots.txt is in it's properly place too.Check out the sitemap.xml and robots.txt file on this site. Just place it in the domain name at the end, http://www.usingwp.com/sitemap.xml and http://www.usingwp.com/robots.txtHope that helps!Frank The robots.txt file and the sitemap.xml work hand in hand. I do believe that you need the robots.txt in the main directory but I’d double check that. Just be sure that the sitemap.xml that is called in the robots.txt is in it’s properly place too.

Check out the sitemap.xml and robots.txt file on this site. Just place it in the domain name at the end, http://www.usingwp.com/sitemap.xml and http://www.usingwp.com/robots.txt

Hope that helps!

Frank

]]> By: retrohttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-383 retro Fri, 05 Mar 2010 22:19:59 +0000 http://www.usingwp.com/?p=61#comment-383 Hey guy, I have a question, this robots.txt business' got me all confused, see I installed wordpress on a separate directory but moved the index.php file to my main index, do I still need the dynamic robots.txt in the main directory or should I just rely on the virtual one in the blog's directory? Hey guy, I have a question, this robots.txt business’ got me all confused, see I installed wordpress on a separate directory but moved the index.php file to my main index, do I still need the dynamic robots.txt in the main directory or should I just rely on the virtual one in the blog’s directory?

]]> By: fthomashttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-317 fthomas Wed, 17 Feb 2010 17:40:56 +0000 http://www.usingwp.com/?p=61#comment-317 Hi Lisa,I've sent you off an email.Frank Hi Lisa,

I’ve sent you off an email.

Frank

]]> By: Lisahttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-316 Lisa Wed, 17 Feb 2010 17:37:54 +0000 http://www.usingwp.com/?p=61#comment-316 I am having that problem right now, google hasn't properly crawled my site yet. I will uncheck it and should I remove the robots.txt file I created with my webserver? I am having that problem right now, google hasn’t properly crawled my site yet. I will uncheck it and should I remove the robots.txt file I created with my webserver?

]]> By: LoveGuruhttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-48 LoveGuru Tue, 21 Jul 2009 10:42:21 +0000 http://www.usingwp.com/?p=61#comment-48 Hi, thanks for the link! I was looking for a plugin like this. Hi, thanks for the link! I was looking for a plugin like this.

]]> By: fthomashttp://www.usingwp.com/plugins/dealing-with-wordpress-and-the-robotstxt-file/comment-page-1/#comment-32 fthomas Mon, 22 Jun 2009 03:13:28 +0000 http://www.usingwp.com/?p=61#comment-32 Great question Pat,You will want to uncheck that setting. Simply take a look at the listing that the robots.txt will contain from within the plugin's page in the admin pages. You will see that it already has the sitemap location specified.Also double check to make sure that there is not 'real' robots.txt present. If there is, simply rename it to robots.txt.old and see if that corrects the problem.Frank Great question Pat,

You will want to uncheck that setting. Simply take a look at the listing that the robots.txt will contain from within the plugin’s page in the admin pages. You will see that it already has the sitemap location specified.

Also double check to make sure that there is not ‘real’ robots.txt present. If there is, simply rename it to robots.txt.old and see if that corrects the problem.

Frank

]]>