# SR 2/2/06 - Added directive to slow down Kosmix (Voyager bot) spider # on page request every 2 seconds. # Wildcarded to slow down other bots that support this directive. # RT 2/6/06 - Added "disallow" paths to prevent Voyager from # crawling non AP=1 pages. Also increased the Crawl-dealy to 10 per Jack # RT 2/9/06 - Added "Allow" paths for voyager to crawl only ap=1 # RT 05/04/2007 - Added section to block iMedixBot User-agent: * Crawl-delay: 2 Disallow: /siteimages/ Disallow: /images/ Disallow: /external/ Disallow: /ads/ Disallow: /printer/ Disallow: /noindex/ User-agent: voyager Crawl-delay: 10 Disallow: /siteimages/ Disallow: /images/ Disallow: /external/ Disallow: /ads/ Disallow: /printer/ Disallow: /noindex/ Disallow: /news/ Disallow: /channel/ Disallow: /subchannel/ Disallow: /ency/ Disallow: /video/ Disallow: /rxdetail/ Allow: /news/1/ Allow: /channel/1/ Allow: /subchannel/1/ Allow: /ency/1/ Allow: /video/1/ Allow: /rxdetail/1/ User-agent: iMedixBot Disallow: /