computer, internet, programming, Microsoft Dynamics, Oracle, Java, J2EE, EJB, SAP, ecommerce strategies, hightech jobs, h1b, web design, MS SQL Server, reporting, customizations, software, ERP, MRP, accounting systems, CRM - popular articles
What is the Robot Text File?
You may have created a personnel page for company employees that you don't want listed. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file.
How do I use it?
You need to upload it to the root of your web site or it will not work - if you don't have access to the root then you will need to use a Meta tag to disallow access. You need to include both the user agent and a file or folder to disallow.
What does it look like?
It's really nothing more than a "Notepad" type .txt file named "robots.txt"
The basic syntax is
User-agent: spiders name here
Disallow:/ filename here
If you use
User-agent: *
The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages.
To disallow an entire directory use
Disallow:/mydirectory/
To disallow an individual file use
Disallow:/file.htm
You have to use a separate line for each disallow. You cannot for example use
Disallow:/file1.htm,file2.html
You should use
Use-agent/*
Disallow:/file1.htm
Disallow:/file2.htm
For a list of spider names visit http://www.robotstxt.org/wc/active/html/ Make sure you use the right syntax if you don't it will not work. You can check you syntax here http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
For help on creating robot text files there is a program call robogen. There is a free version and an advanced version, which costs $12.99 http://www.rietta.com/robogen/
Alba Spectrum popular articles series: FAQ, Reviews, Introductions, Product Selections, Advises, Definitions, online marketing
We are serving wholesale & retail customers in Illinois, California, Texas, Wisconsin, New York, Washington, Ohio, Michigan, Indiana, Arizona, New Mexico, Louisiana, Florida, Georgia, Minnesota, Utah, Virginia, Georgia, Hawaii, Iowa, Colorado, Ontario, Quebec, Alberta, British Colombia. We also serve customer internationally in New Zealand, Europe: UK, France, Poland, Italy, Germany, Russia, India, Byrma, Thailand, Holland, Denmark, Sweden, Norway, Indonesia, Austria, New Zealand, Pakistan, Afghanistan, Iran, Spain, Argentina, Brazil, Chile, Uruguay, Paraguay, Peru, Equador, Colombia, Venezuela, Panama, Costa Rica, Canada, South Africa, Nigeria, Portugal, Greece, Turkey, Asia: India, China, Philippines, South Korea, plus business metros: Chicago, Los Angeles, Phoenix, Boston, Atlanta, Minneapolis, Fargo, Seattle, Miami, Orlando, Detroit, Buffalo, Toronto, Paris, London, Montreal, Denver, Warsaw, Berlin, Prague, Rome, Karachi, Sao Paulo, Rio de Janeiro, Moscow, Buenos Aires, Dehli, Mumbai, Beigin, Cairo, San Francisco, Fremont, Naperville, Oakland, Melburn, Sidney, Sent Petersburg, Tampa, New Orleans, Houston, Dallas, Mexico City, Bogota, Caracas, Lima, Salvador, Recife, Brasilia, Curitiba, Goiania. http://www.albaspectrum.com