top of page
Post: Blog2_Post
  • Writer's pictureDan Lauer

Latest Google Video - How Google Search Crawls Pages

The latest episode from Google's Gary Illyes is here on YouTube! And let me tell's a great one....and I think really interesting and important.

The latest episode Gary Illyes explains "How Google Search Crawls Pages."

In the episode Gary explains basically how Google crawls websites, finds URLs, and the whole process of crawling, fetching, and rendering.

What the SEO content goblins find most interesting is...

  1. The importance of Google discovering URLs and indexing them, because other pages, like category pages link to them.

  2. Really stressing the importance of JavaScript and making sure things load and render properly for 'google-bot'.

  3. Lastly, while Gary reiterated that XML Sitemaps are NOT required - they really do help make the process easier.

The SEO content goblins really find it interesting some of details on these three points. Google is not always transparent, but when Google mentions something related to crawling and indexing - your SEO Ears should perk up - and you should make sure that you double-check your site's performance.

When the SEO content Goblins conduct a full-site Technical Audit - one of things we stress and look at is how well is your site being indexed by Google. Gary mentions in the video, and this is nothing new, there are trillions of web pages out there. Google does not crawl and index everything. So - anything that you can do to make sure your site easily crawled and understood by Google, the better. Not guaranteeing rankings - you still need great, 'helpful content', but don't make it harder for you, and Google, than it needs to be.


This video highlights the importance of some basic and fundamental SEO topics:

  1. Interlinking strategy in your site - don't create page that nothing links to - Google is going to have a hard time discovering them.

  2. Crawl speed - don't have a slow website - if it takes Google a long time to crawl because of inefficient html, css, javascript coding, its the same for users - Google may start to crawl less, or just not index that page at all - this point just hammers home page load speed and core web vitals - do better for your users - specifically around images, and clean code - make sure the code you have on the page, none of those resources are blocked from google crawling.

  3. Server errors - make sure you don't have a ton of unnecessary broken links.

  4. Only have publicly accessibly pages crawlable to Google - basically don't have your authenticated, login pages marked index, follow, included in sitemaps, etc.

  5. Make sure your JavaScript is clean - loads fast - not blocked again - let Google-Bot efficiently crawl and render your page.

  6. Sitemaps - not required, but man do they really help with efficiently with Google. They still might find your pages without an XML sitemap, but your making Google's job harder - stop it! - Have a Good - optimized XML sitemap - make sure all URLs in the sitemap are working, not broken, crawlable, not marked no index, not blocked with robots.txt file, include last modified data and other meta data. Google probably looks at this first, to find the most updated, new pages and crawls those. Can you still get indexed without a sitemap - sure, is this a magic silver bullet no - but in the aggregate it will help if this is 100% working order.

This episode was really intriguing, not so much how Google crawls, renders, fetches, etc. but more so the topics under each one of those. The SEO Content goblins really loves what Google is doing here and hope they keep on doing. This should help ease some of the bad blood between SEO's, website owners, and Google.

The next episode it looks like Gary will discuss how Google makes pages available through search - this one should be a doozy. The SEO Content Goblins can't wait for Episode 3!

9 views0 comments


bottom of page