Googel crawl stats report

So, Google today launched an all new update to it’s ever-evolving Search Console product. A brand new “Crawl Stats Report” that promises SEOs more date and insights. How can SEOs make use of it. Let’s take a look.

Here’s the official tweet from Google Search Central.

What is Google Crawl Stats Report
The Crawl Stats report shows you statistics about Google’s crawling history on your website. For instance, how many requests were made and when, what your server response was, and any availability issues encountered. You can use this report to detect whether Google encounters serving problems when crawling your site.

So, looks like it’s an amazing tool for SEOs to find out what’s wrong with your website. And of course, help fix those things that’s causing Google not to crawl your website properly.

For beginner SEOs, let’s take a step back.

What is search engine crawling?

Crawling is first out of the three primary functions of search engines. The functions being Crawling, Indexing and Ranking.

  1. Crawling: This is the first step where a search engine discovers a new link/URL from a website and scans the content.
  2. Indexing: In this step, a search engine stores and organizes the content discovered during the crawling process.
  3. Rank: In this step, a search engines chooses parts of the content indexed that matches a search query in an ordered list format (Most relevant to least relevant).

What does Crawl Stats show?

As of now, crawl stats provides the following stats.

  • Total crawl requests to the entire website in the chosen timespan.
  • Total downloaded size in Bytes.
  • Average response time in Microseconds.
Total crawl requests
Total number of crawl requests to your site, in the time span shown. Duplicate requests to the same URL are counted.
Total downloaded size
Total size of all files and resources downloaded during crawling, during the period shown. If a resource is already cached from a previous crawl it is not counted. Bytes include file HTML, associated image bytes, script files, and CSS file.
Average page response time
Average page response time for a crawl request to retrieve the page content. Does not include retrieving page resources (scripts, images, and other linked or embedded content) or page rendering time.

Source – Google

Google Search Console will also show us the child domains in your website and the issues associated with them, if any.

Clicking on the items will reveal the details.

It also shows the crawl requests breakdown by response, file type, purpose and Google bot type.

Crawl Stats by File Type

This metric shows the split between different file types and if there are any unusual issues that need attention.

Also, this is an excellent way to figure out if there are issues with any particular file type you’re hosting, that’s causing a larger issue – like page speed and load times.

For example, if there are several/any JS files that is taking an unnatural amount of time to crawl. Perhaps from a third part site like (Facebook or Google AdSense).

Click through on each file type to narrow down on the issue.

Crawl Stats by Response

Crawl stats by response is probably the best way to find out if there are common issues with your website.

Like 404 errors (broken links) for example.

It shows five types of responses. From OK (200), Not Modified (304), Moved Permanently (301), Not Found (404) to Page could not be reached.

A sudden spike in any of these could mean there’s an issues gone undetected and you need to fix it asap.

Nice!

To summarise, here’s how SEOs can make use of Google’s Crawl Stats report.

  1. Evaluate site health check in one glance.
  2. Find out what resources are causing page load issues. (using Crawl Stats by File Type)
  3. Find out issues with broken links, 301 redirects etc (using Crawl Stats by Response)
  4. Find out health issues of child/sub domains separately.
  5. Find out if any part of website is unavailable for crawling. (using Host Status)

I think Crawl Stats is going to be the go-to place for SEOs to do a reliable site health check, without using third party tools.

It’s nice to see Google Search Central team come up with these useful updates. Long time coming!

+ posts

Can talk about SEO and Content Marketing all day long. Been helping companies sell stuff online for the last fifteen years, in the US, UAE and India. Rankz fan. Helps with Growth Marketing at Rankz.

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
Tweet
Share
Share
Pin