XMPP JavaScript Library

How to stop Google news crawler from indexing web site or a page.

UnCategorised, Sachin Puri, 2010-09-17 13:45:17

Blocking Goolge crawler using "robots.txt" file

To stop google news crawler from indexing your site add following lines in "robots.txt" file.

User-agent: Googlebot-news

Disallow: /

This will block your entire web site.

 

To block specific article/ page use following:

User-agent: Googlebot-news

Disallow: /<path>/<page-name>.<ext>

Note: Replace <path> with path of article and <page-name> with name of page and <ext> with extension of that page for example .html . php .htm .asp ect

 

 

Blocking Goolge crawler using "<meta-tags>"

Add following meta tag in <head></head> section of page.

<META NAME="Googlebot-news" CONTENT="noindex" />

This will tell Google crawler that you don't want to include this article in google news.

Add Your Comment
   
    Yes! I want to receive all comments by email

  by Manik on 20-Sep-2010 07:13 pm
Nice Thanks
  • Reply
  •  2 Like
  •  2 Dislike
  • Report