Search Engine Crawling Updates in the Coming Years According to Fabrice Canel
Indexing and Crawling: what you should know September 23, 2021
Fabrice Canel is a principal program manager at Bing, Microsoft. In this video interview with Jason Barnard, he talks about the adoption of APIs in content management systems and the evolution of search engine crawlers in the coming years.
00:00 Introducing, Fabrice Canel
00:07 The future of Search Engine Crawling
01:15 CMS adopting APIs
Transcript
So I do believe that the Future is within the next 2 to 3 years, we will see really content management systems adopting APIs, but this is for Bing or other search engine to really help them to understand the latest of the internet.
Because I get the feeling with API, “I don’t want to bother Bing or Google by submitting this page ’cause I just change one character”, but actually relatively speaking rather than actually forcing you to crawl it and figure it out for yourselves. That’s quite a favor I’m doing you.
Yeah, this is a favor. Again, you will always be smarter than search engines. Oh! Thank you! Me, personally…
Because you know what you do with your site, you know when you write content, we don’t. We can try to figure it out. We can have machine learning really optimized for Jason’s website. Okay, maybe we will not do this because it will cost really a lot of money to try to be as smart as you š But in the end, you know more than me. I believe really there is opportunity for the industry to change the way we are doing this (crawling).
I do believe in a kind of notification to search engines (using an API), to help with having the latest content indexed and to help reduce greenhouse (gas emissions)… reduce ongoing crawling all this content that really does not matter for the business. So I do believe in this direction, we are working on easing this ability for website content management systems to notify us and other search engines.
And at some point I do believe that maybe half of the industry within the next two to five years, will be moving to that, if not more. There will be always 20% of sites that are not adopting this API. This is still fine. We will continue improving machine learning, but yeah, this is the future and we are transforming little by little the industry towards this model.
