robots.txt : Java Glossary

go to home page R words local find full screen, hide local find menu Google search web for more information on this topic jump to foot of page translate this page with Babelfish punctuation 0-9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z (all) ©1996-2009 Roedy Green, Canadian Mind Products
robots.txt
robots.txt is a file you can place in the root directory of your website to tell web crawlers (search engines) which pages to index and which to ignore. A typical robots.txt file might look like this:
# parts of the mindprod.com website not indexed
user-agent: *
disallow: /template.html
disallow: /include/
disallow: /jgloss/include/
It means, for all browsers, don’t look at the file template.html or anything in the two directories mentioned. There is no way to tell it to avoid certain file extensions.
clockwatchers.com robots.txt tutorial
Google sitemap
Google Sitemap Generator Utility
outfront.net robots.txt tutorial
Robot
robots.txt howto
search engines
spider
CMP homejump to top You can get the freshest copy of this page from: or possibly from your local J: drive (Java virtual drive/mindprod.com website mirror)
http://mindprod.com/jgloss/robotstxt.html J:\mindprod\jgloss\robotstxt.html
CMP logofeedback Please email your feedback for publication, errors, omissions, typos, formatting errors, ambiguities, unclear wording, broken/redirected link reports, suggestions to improve this page or comments to Roedy Green : feedback email
mindprod.com IP:[65.110.21.43]
view BlogYour face IP:[38.107.191.106]
You are visitor number 8,705.