About /robots.txtIn a nutshellWeb site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Quelle: The Web Robots Pages