Apr 14, 2011

What is ROBOTS.TXT

Seo noida
ROBOTS.TXT:- Crawlers,Robots,Spiders etc are all recognized with the same. These crawlers indexes our website or Blog in their search engine databases(or u can say in index).
Google robots :- Googlebot,
Yahoo robots :- Yahoo Slurp
MSN robots :- MSN bot.

robots.txt is nothing it is only a text file which is created at the time of creating a website.This name is case sensitive, maintain all the letters to be lower case.


In the robots .text We give the instruction to the search engine through this file robots.txt file.We place this file robots.txt in the root directory of the site. That means we place this file besides the index file of our website.


How to create a robots.txt file :-
User-agent: *
Disallow: /on-pageseo_folder/on-pageseo.html

user agent: * means that the instruction is for all the search engine (crawlers).
Disallow: instructs the crawler not to index the file /on-pageseo_folder/on-pageseo.html
If u want that my all file are not index then use this Coading
Disallow: /on-pageseo_folder

2 comments:

  1. Great article. I'm currently having an issue where a lot of pages on a website cannot show AdSense because the robot can't read the pages.

    ReplyDelete
  2. Heya¡­my very first comment on your site. ,I have been reading your blog for a while and thought I would completely pop in and drop a friendly note. . It is great stuff indeed. I also wanted to ask..is there a way to subscribe to your site via email?
    India SEO Services

    ReplyDelete