Ok, my 2 cents:
robots.txt is a convention. robots.txt is not a way of protecting your website, it's just a way of telling bots how you'd like them to behave. but there's no guarantee that they'll obey it.

If you want to stop bad bots, use server-side scripting. check the user agent, respond with 404 if you don't like it. but even then, some bad bots will identify themselves as a browser, so you'll never know....