Fabrice Canel Explains How BingBot and GoogleBot Choose pages to Crawl on the Web
The Low-down on IndexNow From Mr Bingbot 02 November 2021 at 18 H CEST (Paris)
Fabrice Canel is aĀ Principal Program Manager at Bing, MicrosoftĀ leading the crawling, processing and indexing team. In this video interview with Jason Barnard, he explains how web crawlers are guided by domains and key pages.
00:00 Fabrice Canel – Mr. Bingbot
00:19 Key pages
00:27 What will happen if we focus on key pages?
Transcript
Would you say most content on the web is not useful or is that exaggerating? I think it’s a little bit exaggerated. We are guided by key pages that are important on the internet and we follow links to understand what’s next.
And if we really focus on these key domains (key pages), then this is guiding us to quality content. So the view that we have of the internet is not to go deep forever and crawl useless content. It’s obviously to keep the index fresh and comprehensive, containing all of the most relevant content on the web.