Web development - Document
robots.txt
A file that gives web crawlers instructions about which parts of a website they should or should not access
What's it for?
To guide search engines and other automated crawlers around a site.
For example…
A website’s robots.txt tells search engines not to scan its private test area while allowing them to scan public pages