Have a look here for more info: http://www.robotstxt.org/wc/exclusion.html.