A Robots.txt file is just a text file that sits on your site and tells the search engine bots to ignore certain folders, such as the admin of your Content Management System. The point of a robots.txt file is to prevent certain files appearing in search results however, search engine don’t have to comply,and the file is readable by anyone so never use it to hide important or privileged information.
A robots.txt file that says the following, would be telling search engines to ignore the entire website:
User-agent: * Disallow: /