#This is a file retrieved by webwalkers a.k.a. spiders that #conform to a defacto standard. # See <URL:http://info.webcrawler.com/mak/projects/robots/robots.html> # The webmaster for this site is: # <name>httpd</name> # <address>httpd@smith.tele.kotnet.org/address> # #Format is: # User-agent: <name of spider> # Disallow: <nothing> | <path> # Flag Date By Reason # $l1- 980130 epc finally understood what the file was for! # #----------------------------------------------------------------------------- # following prevents access to /cgi-bin/* for spiders, allow /cgi-bin.txt User-Agent: * Disallow: /cgi-bin/