Robots Exclusions
This video explains custom robots files and how to create them. Robots files are used to provide web crawlers instructions on what to exclude from searches. You can either append a robots file to the default robots file or replace it.