So, Google today launched an all new update to it’s ever-evolving Search Console product. A brand new “Crawl Stats Report” that promises SEOs more date and insights. How can SEOs make use of it. Let’s take a look.
Here’s the official tweet from Google Search Central.
So, looks like it’s an amazing tool for SEOs to find out what’s wrong with your website. And of course, help fix those things that’s causing Google not to crawl your website properly.
For beginner SEOs, let’s take a step back.
What is search engine crawling?
Crawling is first out of the three primary functions of search engines. The functions being Crawling, Indexing and Ranking.
- Crawling: This is the first step where a search engine discovers a new link/URL from a website and scans the content.
- Indexing: In this step, a search engine stores and organizes the content discovered during the crawling process.
- Rank: In this step, a search engines chooses parts of the content indexed that matches a search query in an ordered list format (Most relevant to least relevant).
What does Crawl Stats show?
As of now, crawl stats provides the following stats.
- Total crawl requests to the entire website in the chosen timespan.
- Total downloaded size in Bytes.
- Average response time in Microseconds.
Source – Google
Google Search Console will also show us the child domains in your website and the issues associated with them, if any.
Clicking on the items will reveal the details.
It also shows the crawl requests breakdown by response, file type, purpose and Google bot type.
Crawl Stats by File Type
This metric shows the split between different file types and if there are any unusual issues that need attention.
Also, this is an excellent way to figure out if there are issues with any particular file type you’re hosting, that’s causing a larger issue – like page speed and load times.
For example, if there are several/any JS files that is taking an unnatural amount of time to crawl. Perhaps from a third part site like (Facebook or Google AdSense).
Click through on each file type to narrow down on the issue.
Crawl Stats by Response
Crawl stats by response is probably the best way to find out if there are common issues with your website.
Like 404 errors (broken links) for example.
It shows five types of responses. From OK (200), Not Modified (304), Moved Permanently (301), Not Found (404) to Page could not be reached.
A sudden spike in any of these could mean there’s an issues gone undetected and you need to fix it asap.
I think Crawl Stats is going to be the go-to place for SEOs to do a reliable site health check, without using third party tools.
It’s nice to see Google Search Central team come up with these useful updates. Long time coming!