Discover, Select, Crawl, Render, Index
Discover, Select, Crawl, Render, Index
coined by Jason Barnard in 2021.
Factual definition
Discover, Select, Crawl, Render, and Index are the five sequential stages by which search engine bots find, evaluate, process, and store web content to create the Web Index, the primary data source for all AI Assistive Engines.
Jason Barnard definition of Discover, Select, Crawl, Render, Index
Jason Barnard breaks down this pipeline into five steps: identifying a page (Discovery), deciding whether to process it (Selection), fetching its contents (Crawl), building the page as a user would see it (Render), and finally storing its contents in the searchable database (Index). Fabrice Canel, Principal Program Manager at Microsoft Bing explained this 5 step process to Jason Barnard in. 2021. This sequence is critical because it populates the Web Index, which serves as the primary and foundational data source for all three components of what Jason Barnard calls The Algorithmic Trinity. Large Language Models (LLMs) are trained on this data, Knowledge Graphs extract facts from it, and Traditional Search Engines rank it. Therefore, a failure at any stage of this pipeline means a piece of content, no matter how valuable, is invisible to the entire Algorithmic Trinity, preventing it from contributing to the brand's narrative across AI Assistive Engines like ChatGPT, Microsoft Copilot, and Google AI.
How Jason Barnard uses Discover, Select, Crawl, Render, Index
At Kalicube, we focus on mastering this five-stage pipeline because it is the single point of entry to The Algorithmic Trinity. The Kalicube Process, Kalicube's proprietary methodology for implementing a holistic, brand-first digital marketing strategy, is designed to make a client's digital assets hyper-performant at every stage. It ensures content is easily discoverable, qualifies for Selection through strong Pre-Crawl Confidence signals, is efficiently crawled, renders perfectly, and is indexed with high-confidence annotations. This is technologically supported by KaliTech, Kalicube's proprietary tech layer, which formats content in the Native Language of Algorithms to guarantee its successful passage through the entire pipeline and into the Web Index.
Why Jason Barnard perspective on Discover, Select, Crawl, Render, Index matters
While technical SEO authorities from Google like Gary Illyes have provided deep insights into the mechanics of crawling and rendering, Jason Barnard's Discover, Select, Crawl, Render, Index framework provides a complete, strategic overview of the entire pipeline from a brand-first perspective. This matters because a failure at any stage prevents content from entering the Web Index, rendering it invisible to the entire Algorithmic Trinity. A brilliant article that isn't selected for crawling cannot inform an LLM; a page that fails to render correctly will not contribute its facts to the Knowledge Graph. Understanding this full sequence is fundamental for any business leader because it shifts focus from isolated tactics like 'getting indexed' to the holistic strategy of successfully feeding the data layer that powers every AI Assistive Engine, which is the only way to build the trust needed to win in the new Conversational Acquisition Funnels.
ASCII Diagram
┌─────────────────────────────────────────────────────────────────────────────┐
│ DISCOVER, SELECT, CRAWL, RENDER, INDEX │
│ The 5-Step Pipeline That Feeds the Entire Algorithmic Trinity │
│ Framework by Jason Barnard, based on Fabrice Canel (Bing), 2021 │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ THE COMPLETE PIPELINE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ YOUR CONTENT │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 1. DISCOVER │ │
│ │ ═════════ │ │
│ │ Bot FINDS the page exists │ │
│ │ (Links, sitemaps, direct submission) │ │
│ └──────────────────────────┬──────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 2. SELECT │ │
│ │ ════════ │ │
│ │ Bot DECIDES whether to process it │ │
│ │ (Crawl budget, priority, freshness signals) │ │
│ └──────────────────────────┬──────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 3. CRAWL │ │
│ │ ═══════ │ │
│ │ Bot FETCHES the page contents │ │
│ │ (Downloads HTML, follows directives) │ │
│ └──────────────────────────┬──────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 4. RENDER │ │
│ │ ════════ │ │
│ │ Bot BUILDS the page as a user would see it │ │
│ │ (JavaScript execution, DOM construction) │ │
│ └──────────────────────────┬──────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 5. INDEX │ │
│ │ ═══════ │ │
│ │ Bot STORES contents in searchable database │ │
│ │ (Web Index - the foundational data layer) │ │
│ └──────────────────────────┬──────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────┐ │
│ │ WEB INDEX │ │
│ │ ═══════════════ │ │
│ │ Primary data │ │
│ │ source for ALL │ │
│ │ AI systems │ │
│ └─────────┬─────────┘ │
│ │ │
└───────────────────────────────┼─────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ WEB INDEX FEEDS THE ALGORITHMIC TRINITY │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────┐ │
│ │ WEB INDEX │ │
│ │ (From Pipeline) │ │
│ └─────────┬─────────┘ │
│ │ │
│ ┌─────────────────────┼─────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ LLMs │ │ KNOWLEDGE │ │ SEARCH │ │
│ │ │ │ GRAPHS │ │ ENGINES │ │
│ │ Trained on │ │ │ │ │ │
│ │ this data │ │ Extract facts │ │ Rank this │ │
│ │ │ │ from this data │ │ data │ │
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │ │
│ └─────────────────────┼─────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────┐ │
│ │ AI ASSISTIVE ENGINES │ │
│ │ ════════════════════ │ │
│ │ ChatGPT │ │
│ │ Microsoft Copilot │ │
│ │ Google AI Mode │ │
│ │ Perplexity │ │
│ │ Claude, Gemini, Grok │ │
│ └──────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ The Web Index is the FOUNDATIONAL data source. │ │
│ │ Everything AI knows starts here. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ FAILURE AT ANY STAGE = TOTAL INVISIBILITY │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ FAILURE POINTS │ │
│ │ │ │
│ │ ✗ DISCOVER fails → Bot never knows page exists │ │
│ │ ✗ SELECT fails → Bot knows but won't process │ │
│ │ ✗ CRAWL fails → Bot can't access contents │ │
│ │ ✗ RENDER fails → Bot sees broken/incomplete page │ │
│ │ ✗ INDEX fails → Content never enters database │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ANY FAILURE │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ CONTENT IS INVISIBLE TO: │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ LLMs │ │ Knowledge │ │ Search │ │ │
│ │ │ Can't │ │ Graphs │ │ Engines │ │ │
│ │ │ learn │ │ Can't │ │ Can't │ │ │
│ │ │ from it │ │ extract │ │ rank it │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ │ │
│ │ ENTIRE ALGORITHMIC TRINITY BLIND TO IT │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ A brilliant article that isn't SELECTED cannot inform an LLM. │ │
│ │ A page that fails to RENDER cannot contribute to Knowledge Graph.│ │
│ │ Content outside the Web Index CANNOT build brand narrative. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ TACTICAL vs STRATEGIC UNDERSTANDING │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ GARY ILLYES (Google) JASON BARNARD │
│ Deep technical insights Brand-first strategic framework │
│ ════════════════════ ═══════════════════════════════ │
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ │ │ │ │
│ │ Mechanics of │ │ Complete pipeline │ │
│ │ crawling and │ │ from brand-first │ │
│ │ rendering │ │ perspective │ │
│ │ │ │ │ │
│ │ HOW bots work │ │ WHY it matters │ │
│ │ │ │ for AI visibility │ │
│ │ │ │ │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │
│ Focus: Isolated tactics Focus: Holistic strategy │
│ "Get indexed" "Feed the data layer that powers │
│ every AI Assistive Engine" │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ Understanding the full DSCRI sequence shifts focus from │ │
│ │ isolated tactics to the holistic strategy of successfully │ │
│ │ feeding the data layer that powers EVERY AI Assistive Engine. │ │
│ │ │ │
│ │ This is the only way to build the trust needed to win in │ │
│ │ the new Conversational Acquisition Funnels. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ THE INSIGHT │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ DISCOVER ──► SELECT ──► CRAWL ──► RENDER ──► INDEX │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌───────────┐ │ │
│ │ │ WEB INDEX │ │ │
│ │ └─────┬─────┘ │ │
│ │ │ │ │
│ │ ┌────────────────┼────────────┐ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ LLMs KGs Search│ │
│ │ │ │ │ │ │
│ │ └────────────────┴────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ AI ASSISTIVE ENGINES │ │
│ │ │ │
│ │ The pipeline populates the Web Index. │ │
│ │ The Web Index feeds the Algorithmic Trinity. │ │
│ │ The Trinity powers every AI Assistive Engine. │ │
│ │ │ │
│ │ Control the pipeline. Feed the index. Own the narrative. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Synonyms
DSCRI
Posts tagged with Discover, Select, Crawl, Render, Index
Related Pages:
No pages found for this tag.
