## COLLAGE SELECT Component: ## Be sure to update each defined SELECT case, including the default one. ## ## The list of specific folder disallows follows the Collage Select ## component and each select case body should end with the User-agent ## specification that allows crawling except for the final list of specific ## disallows. # # for all compliant web bots... # disallow access to all dynamically generated content as well as # access to static, non-informational and deprecated document folders. # # known malevalent, but compliant crawlers: # # this is the robots.txt file that will appear in production User-agent: * Disallow: /OTS Disallow: /acquisition Disallow: /catalog Disallow: /custserv Disallow: /dev Disallow: /documents Disallow: /genetics Disallow: /icc Disallow: /images Disallow: /includes Disallow: /mmrrc Disallow: /mmrrcSearch Disallow: /project Disallow: /obsolete Disallow: /survey Disallow: /tracking Disallow: /Prototypes Disallow: /Templates Disallow: /cou