Magento Forum - UK Magento Forum (Unofficial)

An unofficial but dedicated Magento Forum for Magento Users, Designers and Developers

Require Robot.txt

A forum to discuss Magento related SEO issues and for SEO professionals working on Magento

Require Robot.txt

Postby Suji » Sun May 29, 2011 3:32 am

Hi Experts,

Can I have Robots.txt file of magento? We have placed robots in google webmaster tool however I need to check what are the things we can allow and disallow.

Please send me the file.

Regards,

Suji
Suji
 
Posts: 42
Joined: Fri Dec 10, 2010 10:09 am

Re: Require Robot.txt

Postby Dx3Webs » Sun May 29, 2011 9:20 am

Dx3webs
UK based Optimised Magento Managed Hosting : Full Managed service with Unlimited technical support
Contact Us | Dx3webs Magento Demo Store
User avatar
Dx3Webs
 
Posts: 305
Joined: Wed Jul 14, 2010 11:04 am
Location: Lincoln, UK

Re: Require Robot.txt

Postby Suji » Wed Jun 01, 2011 8:24 am

Thanks Dx3Webs... This is sufficient for me.

regards

Suji
Suji
 
Posts: 42
Joined: Fri Dec 10, 2010 10:09 am

Re: Require Robot.txt

Postby Davis Joseph » Thu Apr 12, 2012 5:47 am

Robots txt is used to exclude content from the search engine spiders / bots. Robots.txt is also called the Robots Exclusion Protocol.
Cheap SSL at just $7 at SSLMatrix.com
User avatar
Davis Joseph
 
Posts: 1
Joined: Thu Apr 12, 2012 5:31 am
Location: USA

Re: Require Robot.txt

Postby robocarlo » Wed Feb 20, 2013 7:26 am

You have install magento Robots.txt extension.
Magento FreeLancer - Magento Custom Developer Creative Themes, FME
robocarlo
 
Posts: 14
Joined: Mon Feb 04, 2013 9:53 am

Re: Require Robot.txt

Postby robocarlo » Thu Mar 14, 2013 9:19 am

Herer is a practical example of robots for very popular magento website,

http://fmeextensions.com/robots.txt
Magento FreeLancer - Magento Custom Developer Creative Themes, FME
robocarlo
 
Posts: 14
Joined: Mon Feb 04, 2013 9:53 am

Re: Require Robot.txt

Postby MageWorx » Tue Sep 10, 2013 1:09 pm

Hello,

Here is another example of the robots.txt file for Magento:
Code: Select all
# Crawlers Setup
User-agent: *

# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt

# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=
Magento Extensions: SEO extension for Magento
User avatar
MageWorx
 
Posts: 5
Joined: Mon Oct 26, 2009 10:16 pm

Re: Require Robot.txt

Postby giovanni.golch » Tue Oct 08, 2013 9:53 am

Well, it's totally different with what i have seen and always been created for the websites. It's nice to see this and learn the magento robot.txt making.
giovanni.golch
 
Posts: 1
Joined: Mon Oct 07, 2013 12:15 pm

Re: Require Robot.txt

Postby brucekenway » Wed Nov 04, 2015 6:12 am

Thank you for your sharing, what a proper answer.
User avatar
brucekenway
 
Posts: 22
Joined: Wed Oct 21, 2015 2:39 am
Location: Dallas, TX


Return to Magento SEO

cron