It means, for all browsers, don’t look at the file template.html or anything in the two directories mentioned. There is no way to tell it to avoid certain file extensions. Note that the Sitemap directive takes a full URL (Uniform Resource Locator), unlike the others.
You can also control spiders with the robots meta tag or with the X-Robots-Tag field in the HTTP (Hypertext Transfer Protocol) response header. and with a sitemap. This is not a human-comprehensible HTML (Hypertext Markup Language) page but a gzipped XML (extensible Markup Language) document in a special format.
This page is posted |
http://mindprod.com/jgloss/robotstxt.html | |
Optional Replicator mirror
|
J:\mindprod\jgloss\robotstxt.html | |
Please read the feedback from other visitors,
or send your own feedback about the site. Contact Roedy. Please feel free to link to this page without explicit permission. | ||
Canadian
Mind
Products
IP:[65.110.21.43] Your face IP:[18.226.166.106] |
| |
Feedback |
You are visitor number | |