Crawling

What is web crawling?

A search engine uses a software called a “web spider” or “web crawling”, which is a type of “bot” (short for “robot”) used to automatically carry out repetitive tasks.

The spider browses the world wide web (WWW), starting with heavily used sites.  It “crawls” from site to site, following links and “reading” the content displayed there.