Web Crawler

A web crawler is an internet bot that systematically accesses the world wide web for the purpose of web indexing (sometimes referred to as spidering) or for other purposes including competitive research or vulnerability scanning. A web crawler is also called a spider, spider bot or shortened to crawler. These crawlers identify themselves to a web server using the User-agent field of an HTTP request. Like a human visitor, a crawler takes up resources when it accesses your website. To limit or prevent unwanted access you should create and configure a robots.txt file.