# This part says that this user agent can not spider anything! User-agent: EmailCollector Disallow: / # The following part allows all other spiders to spider all despite of disallowed folders! User-agent: * Disallow: /_private/ Disallow: /_vti_bin/ Disallow: /_vti_cnf/ Disallow: /_vti_log/ Disallow: /_vti_pvt/ Disallow: /_vti_txt/ Disallow: /cgi-bin/ Disallow: /Templates/ Disallow: /_vti_inf.html Disallow: /clip_image001.gif Disallow: /postinfo.html Disallow: /style.css