By Vinod Kumar ( 26/01/2018 ) For many, Google is the internet. It's the starting point for finding new sites, and is arguably the most important invention since the internet itself. Without search engines, new web content would be inaccessible to the masses. But do you know how search engines work? Every search engine has three main functions: crawling (to discover content), indexing (to track and store content), and retrieval (to fetch relevant content when users query the search engine). Crawling Crawling is where it all begins: the acquisition of data about a website. This involves scanning sites and collecting details about each page: titles, images, keywords, other linked pages, etc. Different crawlers may also look for different details, like page layouts, where advertisements are placed, whether links are crammed in, etc. But how is a website crawled? An automated bot (called a "spider") visits page after page as quickly as possible, using