Magento Forum - UK Magento Forum (Unofficial)

An unofficial but dedicated Magento Forum for Magento Users, Designers and Developers

Restricting bots with robots.txt file

A forum to discuss Magento related SEO issues and for SEO professionals working on Magento

Restricting bots with robots.txt file

Postby Mark » Tue Jan 04, 2011 12:51 pm

Hi,

I've been reading quite a bit about this topic recently, and keep seeing differences of opinions to do with what should and should not be crawlable by the search engine bots.

Mainly, should I allow
/catalogsearch/
including:
/catalogsearch/advanced and /catalogsearch/results to be indexed or not??

Thanks
Mark
 
Posts: 65
Joined: Sat Sep 19, 2009 10:53 pm
Location: UK

Re: Restricting bots with robots.txt file

Postby edmondscommerce » Tue Jan 04, 2011 1:59 pm

I'm of the opinion that search results pages can be great for SEO so I would let the spiders crawl them, and I would put effort into optimsing those pages so that they can become highly optimised for the specific search phrase.
User avatar
edmondscommerce
 
Posts: 1157
Joined: Fri Sep 11, 2009 8:55 am
Location: UK

Re: Restricting bots with robots.txt file

Postby redfeilds18 » Mon Jul 16, 2012 4:01 pm

i think disallowing them is a better option.. it can create duplicate content which is not good for seo.
redfeilds18
 
Posts: 2
Joined: Mon Jul 16, 2012 3:53 pm

Re: Restricting bots with robots.txt file

Postby stevensagaar » Sat Oct 11, 2014 11:06 am

I agree that search pages should be excluded from search engines, here is my bear minimal robots.txt file for all Magento sites- :

##========================================
User-agent: *
## Do not crawl sub category pages that are sorted or filtered.
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*
Disallow: /*?___from_store=*
Disallow: /*?cat=*
Disallow: /*?q=*

## Do not crawl links with session IDs
Disallow: /*?SID=

## Do not crawl checkout and user account pages
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

## Do not crawl search pages and not-SEO catalog links
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product/gallery/
Disallow: /javascript/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

##========================================
stevensagaar
 
Posts: 2
Joined: Sat Oct 11, 2014 10:49 am

Re: Restricting bots with robots.txt file

Postby brucekenway » Wed Oct 28, 2015 6:33 am

stevensagaar wrote:I agree that search pages should be excluded from search engines, here is my bear minimal robots.txt file for all Magento sites- :

##========================================
User-agent: *
## Do not crawl sub category pages that are sorted or filtered.
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*
Disallow: /*?___from_store=*
Disallow: /*?cat=*
Disallow: /*?q=*

## Do not crawl links with session IDs
Disallow: /*?SID=

## Do not crawl checkout and user account pages
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

## Do not crawl search pages and not-SEO catalog links
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product/gallery/
Disallow: /javascript/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

##========================================

I agree with you, your post is informative.
User avatar
brucekenway
 
Posts: 21
Joined: Wed Oct 21, 2015 2:39 am
Location: Dallas, TX

Re: Restricting bots with robots.txt file

Postby MageCloud » Thu Nov 26, 2015 1:34 pm

Actually Google's webmaster guidelines specifically state that search results shouldn't be indexed - they don't add value and create duplicate content.

Here's the post from Google about this: https://support.google.com/webmasters/answer/35769

This is mentioned in "Technical guidelines" paragraph.
MageCloud
 
Posts: 28
Joined: Thu May 07, 2015 9:16 am


Return to Magento SEO

cron