Comments on: Problems With WordPress Posts Going Supplemental In Google’s Index

By: Steven Bradley

Steven Bradley — Thu, 28 May 2009 20:53:35 +0000

First, keep in mind this post is a few years old now and things do change.

I’ve seen people suggesting robots.txt not being the best way to eliminate duplicate content too. There was a video by Matt Cutts a couple years back talking about the different ways Google might find urls on your site. None of the methods for blocking certain pages from getting indexed was 100% effective and Matt suggested trying a few to make sure the page didn’t get crawled.

Some methods would be robots.txt, meta noindex, meta canonical. I think you have to look at what each does in order to decide which is best to use and I think it’s ok to combine techniques.

robots.txt is a quick way to tell search engines not to crawl everything inside a directory for example, but if other sites are linking to those pages then they might still get crawled. Adding noindex should ensure the page doesn’t get indexed.

By: Autocross

Autocross — Tue, 26 May 2009 16:39:19 +0000

Just last night I was looking for a good article on wordpress robots.txt, and now I found it. Last night the best article I found said robots.txt was not the proper way to eliminate duplicate content. The suggested noindex meta tags instead. What are your thoughts on this?

By: Steven Bradley

Steven Bradley — Tue, 29 Jul 2008 22:17:55 +0000

I think what you’re seeing is something different than what I described here.

PR comes from links pointing into your page. It’s also important to note that Google only updates the PR we see in the toolbar every few months. They just updated over the weekend so you may be seeing PR for your blog now.

Google and WordPress have also changed things in regards to the feeds and the supplemental index since I wrote this post. While I still think it important to eliminate duplicate content from your site I don’t know that what I described here is the same problem it was a couple years ago.

By: Stacy

Stacy — Sat, 26 Jul 2008 22:08:36 +0000

Thanks for the post. My problem is my blogger blog. It has been a PR0 forever and I can’t figure out why. Is this different from being filed in supplemental? (I don’t see the supplemental designation when I check but I am pretty new to the techie side of SEO & blogstuffs.)

The funny thing is I have gotten to #1 on a few Google searches yet still no PR value? I guess I have a lot to learn. (I mainly blog for the fun of it.)

By: Wordpress post supplemental on googlebots | Just Another Dang Blog

Wordpress post supplemental on googlebots | Just Another Dang Blog — Mon, 21 Jul 2008 03:05:46 +0000

[...] in the Wordpress post that is being picked up by the googlebots. Nice read about this issue here Problems With WordPress Posts Going Supplemental In Google’s Index. And found a solution by downloading this plugin All In One SEO [...]

By: Best Wordpress Plugins You Must Use

Best Wordpress Plugins You Must Use — Tue, 24 Jul 2007 17:59:43 +0000

[...] Although Wordpress offers many features it lacks in few. Like the duplicate content issue of wordpress, where whole pages of the wordpress blogs were seen to be moving in the Google Supplemental Index. The best remedy for this is the Wordpress All in One SEO Pack! As the name suggests it takes care of everything a search engine may frown upon so you don’t have to worry about anything. [...]

By: Steven Bradley

Steven Bradley — Mon, 14 May 2007 03:57:53 +0000

Glenn I would think the first three lines that didn’t work would still pick up the ones you’re trying now. The /*/ should pick up anything so the commands are just telling Googlebot not to index what anything that ends in feed/ or rss/ or trackback/.

When you do a site: command at Google for your blog and you see the supplemental results are they ending in feed/, rss/, and trackback/? If not then you might be having a different issue.

Feel free to post a link to your blog here or email me if you want. I’ll be happy to take a look.

By: Glenn

Glenn — Mon, 14 May 2007 03:21:54 +0000

I used the following lines in my robots.txt file but did not see any results in a week.
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$

I discovered my feed had the index.php in the URL so I changed the paths as follows. Does anyone know if this is correct? When I type http://www.MyBlog.com/blog/index.php/feed in a browser window I see XML data.

User-agent: Googlebot
Disallow: /blog/index.php/feed/$
Disallow: /blog/index.php/feed/rss/$
Disallow: /blog/index.php/trackback/$
User-agent: *
Disallow: /blog/index.php/wp-
Disallow: /blog/index.php/feed/
Disallow: /blog/index.php/trackback/
Disallow: /blog/index.php/rss/
Disallow: /blog/index.php/comments/feed/
Disallow: /blog/index.php/page/
Disallow: /blog/index.php/date/
Disallow: /blog/index.php/comments/
Disallow: /rsscb

By: Steven Bradley

Steven Bradley — Sat, 12 May 2007 00:53:28 +0000

Good point Colin. I use the same plugin, though in lazy mode where it picks up the first whatever number of characters of the actual post. I noticed an improvement in Yahoo rankings and traffic too shortly after adding it.

By: Colin Carmichael

Colin Carmichael — Fri, 11 May 2007 18:23:38 +0000

One of the other things WP does poorly is unique meta description tags. I just started using this plugin which seems to do a good job of letting me control the description tag of each post & page using the custom fields.
link: http://guff.szub.net/2005/09/01/head-meta-description/