Robots.txt

A file that is placed in the root directory of the site and its main task is to control the behavior of the search engines online. It identifies all the pages and sections of the site that you do not wish to be indexed by the search engines. It may be, for example, user credentials page or pages with any technical or private information.