computer, internet, programming, Microsoft Dynamics, Oracle, Java, J2EE, EJB, SAP, ecommerce strategies, hightech jobs, h1b, web design, MS SQL Server, reporting, customizations, software, ERP, MRP, accounting systems, CRM - popular articles
Robot.txt Good or Bad!
Basically, robots.txt is a plain text file which is placed in a server's root directory it includes information on whether search engine robots should index the site or parts of the site. The file (line begins with '#'), then 'User-agent' lines. Usually, the User-agent line is simply a wildcard, to exclude all robots, like so :
# robots.txt for http://yoursite.com User-agent: *
although you can write seperate agent/disallow sections for different robots. Next comes the Disallow section. this is read by the robot and from there, it determines what's off-limits when it comes to indexing your site.
# robots.txt for http://yoursite.com User-agent: * Disallow: /administration/ # nothing under /administration/ should be spidered Disallow: /temp/ # these are temporary files Disallow: /active.asp # active content here, no point spidering it
Disallowing pages deep into your structure can be good for your users, they won't find themselves halfway through the site with no idea how to get out. Then again, the more search engine entries you have, the better, right? It's up to you to decide what should or shouldn't be excluded.
OK so robot.txt is good for your users and to tell the search engines which pages to list but. Here is the bad part not all robots or bots are good some will ignor the robot.txt file and just index all pages it comes across. So some of your admin pages could get displayed somewhere. Also now you know about robot.txt it needs to be in the root directory what's to stop someone who reads this article just going around and typing http://yoursite.com/robot.txt this would display your robot.txt file!
That would be like going into a pub and leaving you wallet on the bar and going to the loo. What's the chances of it still being there when you get back - yeh slim! So now you know a bit about robot.txt it up to you to decide.
Good luck.
© John Hutchison
This article can be found at http://www.searchhuts.co.uk/portal/articles/activenews_view.asp?articleID=5
Alba Spectrum popular articles series: FAQ, Reviews, Introductions, Product Selections, Advises, Definitions, online marketing
We are serving wholesale & retail customers in Illinois, California, Texas, Wisconsin, New York, Washington, Ohio, Michigan, Indiana, Arizona, New Mexico, Louisiana, Florida, Georgia, Minnesota, Utah, Virginia, Georgia, Hawaii, Iowa, Colorado, Ontario, Quebec, Alberta, British Colombia. We also serve customer internationally in New Zealand, Europe: UK, France, Poland, Italy, Germany, Russia, India, Byrma, Thailand, Holland, Denmark, Sweden, Norway, Indonesia, Austria, New Zealand, Pakistan, Afghanistan, Iran, Spain, Argentina, Brazil, Chile, Uruguay, Paraguay, Peru, Equador, Colombia, Venezuela, Panama, Costa Rica, Canada, South Africa, Nigeria, Portugal, Greece, Turkey, Asia: India, China, Philippines, South Korea, plus business metros: Chicago, Los Angeles, Phoenix, Boston, Atlanta, Minneapolis, Fargo, Seattle, Miami, Orlando, Detroit, Buffalo, Toronto, Paris, London, Montreal, Denver, Warsaw, Berlin, Prague, Rome, Karachi, Sao Paulo, Rio de Janeiro, Moscow, Buenos Aires, Dehli, Mumbai, Beigin, Cairo, San Francisco, Fremont, Naperville, Oakland, Melburn, Sidney, Sent Petersburg, Tampa, New Orleans, Houston, Dallas, Mexico City, Bogota, Caracas, Lima, Salvador, Recife, Brasilia, Curitiba, Goiania. http://www.albaspectrum.com