To prevent undesirable content material from the lookup indexes, webmasters can instruct spiders to not crawl specified information or directories in the common robots.txt file in the root Listing in the area. In addition, a site might be explicitly excluded from a search engine's database by using a meta tag https://www.imdb.com/title/tt30110854/?ref_=wl_t_1