Looking at our site's pages indexed by Google I noticed that lots and lots of pages like: www.site.com/index.asp?PageAction=EMAILFRIEND&ProductID=1024 are indexed by Google and all with the same page title, description, and meta tags.

Can I do something so Google does not pick up these pages? Pretty useless pages and for sure hurting our SE results.

thanks in advance for any tips and help.

Mike

Mike,

Yes, there is a way to prevent engines from crawling information. It should be noted in your robots.txt file
The one below is a good example of what your robots.txt file (perhaps) should look like to exclude such directories/pages.

Refer to: http://www.searchtools.com/robots/robots-txt.html


User-agent: *

Disallow:/admin/
Disallow:/affiliatewiz/
Disallow:/eproducts/
Disallow:/themes/
Disallow:/browseproducts/
Disallow:/BROWSEPRODUCTS/
Disallow:/QUICKORDER/
Disallow:/ADVSEARCH/
Disallow:/EMAILFRIEND/
Disallow:/Custreg/
Disallow:/https:/
Disallow: /index.asp?PageAction=ADDPRODREVIEW

If someone has a better robots.txt file, please post.


Scooter

Thanks for the reply. That was also my first thaought. Your suggested Robots.txt file is more comprehensive than the one we have now and I will definitely use it.

Another question: You know all commerce sites have a duplicate copy of their home page as www.homepage.com/index.asp that is indexed by search engines. This page is returned as a result of 404 page not found error. Can this one be disallowed in robots.txt file?

Mike

Scooter, I also have these,

Disallow:/LOGIN/
Disallow:/CARTDETAILS/

although maybe they are included in your /https:/

Why do you have - Disallow: /index.asp?PageAction=ADDPRODREVIEW
and not just - Disallow: /ADDPRODUCTREVIEW/ ?

Read somewhere if you have a G**gle sitemap, this is a good idea to have at the bottom. Not sure if it's really necessary.

Sitemap: http: // www. sciencestoreforthestars .com/ sitemap.xml - without the spaces of course

Scooter,

Google Webmaster Tools allows you to test changes made to robots.txt before making the actual change. So, in Webmaster Tools I added the :

Disallow: / EMAILFRIEND

and tested to see if Googlebot follows a link to http://www.chipsandgames.com/index.asp?Pag...;ProductID=1024 and it did allow it. It was not blocked.

Then I changed to Robots.txt line to the following and it worked:

Disallow: /index.asp?PageAction=EMAILFRIEND

I tested the same thing with ADDPRODUCTREVIEW and same result.

Testing the same page URL as above confirmed that it was blocked by the above command.

cheers,

Mike

I manually submitted one of the EMAILFRIEND URLs from our site to Google (in Google Webmaster Tools) to be removed from its index and after a couple of hours that it was pending my request was denied. Google gave the following explanation:

"Your request has been denied because the webmaster of the site hasn't applied the appropriate robots.txt file or meta tags to block us from indexing or archiving this page."

I had the following in robots.txt:

Disallow:/EMAILFRIEND/

I will next test it with adding the following:

Disallow:/index.asp?PageAction=EMAILFRIEND

Mike