Jack Lewis notes Google apparently disregarding robots.txt – in fact using robots.txt to actively hunt out information. (hat tip: Little Green Footballs) I question the conclusions so rapidly jumped to, but the behavior is interesting and worth investigation. Just out of curiosity, I set up a honey pot robots.txt on my home domain, so see what pops up. I’ll let you know.
UPDATE (2/18): So far, Google has been honoring the file.
I’ve had a robots.txt file on my domain for years and Google honors it. I just put in a few choice phrases that would normally go right to it, but it’s nowhere to be found in Google.