click Are you confused about XML Sitemaps? You might studied a lot but still not clear on how to deal with these?
No problem, today I will talk about Sitemaps in detail and clear all your misconceptions about the same.
Generally people do not care about Sitemap.When they came to know that is an essential aspect from the SEO point of view, then they get confused on how to proceed?
Let me cover all the possible doubts and their solution in this post. I will explain everything in depth and try to quote a few examples to get you proper implementation for your use case.
What is an XML Sitemap?
The XML sitemap is a tree structure of your website that lists down all the important pages that you want Google to consider for your website. Sites without sitemap often notice that their important pages aren’t available in Google index.
At times there are pages that are relevant and important to the site but are missing any internal link or coverage. Those pages might not get included in Google index. But when you especially include those pages in the xml sitemap they usually are indexed.
This tree-like structure or sitemap may differ for different use cases and various sites. It is different for an e commerce website and different for a blog.
Sometimes a site might have more than 1000 pages but only 400 of those are important, then it is must for a webmaster to include those 400 pages in the sitemap.
The sitemap is a tool available with you to tell Google that these 400 pages are the one that you consider are important and should be included in the Google search.
Why do you need an XML Sitemap?
As discussed above, the answer is really simple. You need to have an XML sitemap because you want to tell Google about the important pages of your website and have a preference for them to be listed in Google search index.
Not having sitemap does not mean that your pages will not be indexed. They will be indexed but you do not have yourself an organized approach to what’s important and what’s not important.
Let me discuss an example here.
Suppose you are having a blog with 1000 pages but when you perform a Google search, you will probably end up seeing that there are only 600 pages indexed. What about other 400 pages? Why are they not indexed? So it might be due to lesser content or no internal links to those pages and on the top, you don’t have a sitemap that could tell Google that some of these pages may be important.
Ideally, we have two kinds of pages associated with our website :
- Regular Content pages: These are freshly prepared content pages for your site visitor and are meant to be there on search results so that you can drive high-quality traffic to these pages.
If you are having the sitemap, you will be able to categorize between what is important and what is not important.
At first, you would not like to include utility pages in your site map and also mark them with “nofollow, noindex” with meta robots or else block them using robots.txt.
Once you are able to list what are the important pages for your website and found that 300 exact pages are highly relevant on this date and must be shown in Google index.
You can now include these 300 pages in the xml sitemap and leave the rest of the pages. Not including remaining pages in sitemap will not ignore those pages to be indexed by Google but will help the 300 pages to be indexed on priority.
I’m repeating again, by including pages in sitemap we tell Google that these pages are important for the site.
In simple words, we need sitemap to make Google identify pages in the sitemap as important and relevant pages for our website visitors.
I would also suggest using an XML sitemap for your website for the following reasons :
- Improve the crawl times: If you have provided a valid sitemap, it would help search engines like Google to crawl your content faster than to crawl individual pages and then the linked pages separately. Usually the later process is scattered approach and have bad crawl time. However if you have a validated sitemap in your website, you will have much improvement in crawl times.
- Prioritized Content: This is the best way to tell Google about which page has most priority and which page has a lesser priority to you. You can attribute priority anything between 0.1 to 1.
- Media sitemap: For websites that have a lot of images and videos, you can have a separate sitemap for your creative media. This way you help your media content to index faster in search results.
- Performance of Pages: Having properly validated sitemap also helps you check the performance of a site as a whole. You will be able to check which page did make to Google SERP and which did not. As you will have your detailed analysis, you will be able to work upon it and further improve the results.
Which pages should be there in the sitemap?
Now that you are clear that we should have important pages in the sitemap. But wait, what’s important?
How do we come to know whether a page is important or not. Now, I would take an example of an e-commerce website so that the relevancy or the importance of a page is clear in terms of including it in the sitemap or not.
An E-commerce website have more than 1000 pages consists of product pages, product categories, subcategories, Seller pages, and utility pages. Now you can define some rules for the sitemap for which pages should be included and which page should not be included.
Not that, if search engine like Google sees the consistency and relevancy of your sitemaps on a regular basis, it starts trusting the website more and thereby increasing the indexing percentage to a higher level.
This categorization of pages can be done dynamically based on specific rules. Although these set of rules will be different for different sites but this example of an e-commerce website, I am defining some rules as examples :
- Product_page.xml: Inclusion of a product page in the sitemap, if it has the product image and the product description more than 400 words. Any product page failing this criterion will not be the part of xml sitemap.
- Category_page.xml: Inclusion of a category page in a sitemap, if the category has more than 5 products and at least 5 of these products are part of product_page.xml sitemap.
This means that any category which has 4 product will not be included in the sitemap. This also implies that if a category has 10 products under it but only 3 of them are the part of product_page.xml sitemap then this category will also not be included in the category_page.xml.
And there can be many such rules depending on what you consider important and relevant for your business.
When we define the sitemap with conditions as listed above, such sitemaps are called dynamic sitemaps. Google usually loves them as they are exact filters to what Google is looking for in a website.
Even for a blog, a sitemap must be split down in 3-4 sitemaps. Like :
Creating specific sitemaps with categorization help you to understand which part of sitemap actually got indexed and which did not.
For example: If you submit a sitemap with 500 pages, Webmaster console will tell the number of pages that were indexed. But it will not list down the pages which were indexed and which were not indexed.
So it is a better practice to split these 500 pages into categories and then submit sitemaps. Doing this will enable you to note which category sitemap have what numbers of non-indexed pages.
Having this knowledge will help you figure out why Google did not include them and what possible changes you can make to get them included in the Google index.
Pro Tip : When you place links to utility pages on the homepage of your site, always remember to make them No follow and No index using meta robots. Since these pages are not important and hence transferring the link juice of the home page to these utility pages is just useless.
How to Inform Search Engines About Your Sitemap?
When it comes to informing about your sitemap, you have to make it live on your website in the form of a web link.
For example: example.com/sitemap.xml. After you are done with making the sitemap live, you can simply go to the search engine and submit your sitemap.
Every search engine has a method to include the sitemap in its indices. There are many search engines whom you can inform about your sitemap like – Google, Bing, Yandex, Yahoo, and many more.
Let me explain to you about submitting a sitemap to major search engines – Google and Bing.
How to submit Sitemap to Google?
You need to have a Google Webmasters account here which you can create at https://www.google.com/webmasters After you are done with the signup process, you simply need to follow these simple steps :
1. Sign in to your Google search console (Webmasters account).
2. You will be asked to select a website in the left sidebar. If you have already added the website there you need to select and if you have not added the website there, you need to first add it in the search console.
3. Now go and click on the “Sitemaps” section in left sidebar to expand it and click on Sitemaps.
4. Now enter the sitemap URL in the sitemap box. Example: https://websitename.com/sitemap.xml
5. Click on Submit button.
Remember, if you have multiple sitemaps then you need to submit all, one at a time. This way you can inform about your website’s sitemaps to Google.
How to Submit Sitemap to Bing?
Just similar to the above process, you first need to have a Bing Webmasters account. So sign up for the one and add your website into it. After you are done with the signup process, you need to follow these simple steps below :
1. Sign in to your Bing Webmasters account.
2. Select the website you just added.
3. In the left sidebar there will be an option for “Indexing Options”, just click over it and then click on sitemap files.
4. Now you just need to enter the sitemap url in the text box. Your sitemap url will look like : https://websitename.com/sitemap.xml
5. Click on Add button and you are finally done adding the sitemap to Bing Webmasters account.
This is how you informed Bing search engine about your website’s sitemap and it can now understand important pages from your website and can index it.
How to Validate XML Sitemap?
If you do not have a correct sitemap and you are a big brand or you have a good website, you are risking a lot of things on stake. Any site must have a valid and working sitemap in a proper acceptable format.
But how to validate the sitemap? There are various free tools available online that will help you validate your sitemap and if there are any problems with the sitemaps these tools will also help you fix those issues. You can have a look at the following sitemap validators :
There are many other sitemap validators and you can choose the one according to your choice and need. However, all the above-mentioned sitemap validators work well.
Common Validation XML Sitemap Errors:
Though there can be many sitemap validation errors and they can be fixed using above sitemap validation tools but few of the common errors which are often found in most cases. These are :
1. At times there are pages which you have already set as “noindex, nofollow” or “noindex, follow” but mistakenly these pages are included in your sitemap.xml. This is a case of contradiction. You must think that you are yourself asking Google to index a page by including it in the sitemap and on the other hand you are using meta robots to not to index it. This confuses Google and other search engines. The same should be fixed.
2. There are few cases wherein pages are blocked using robots.txt, but you have included the same in the sitemap. This is again a case of contradiction wherein you might confuse Google on what you say and what you actually ask Google to do.
5 Best XML sitemap generators
So far we have covered almost everything about sitemap and now we actually need to understand how can we generate a valid and perfect sitemap as per our website’s requirement. I will be mentioning here 5 best XML sitemap generator for WordPress that you can wisely use in your WordPress website.
1. Yoast SEO
It’s free. And the best part of Yoast is that it analyses your website thoroughly and then create the sitemap. Once your sitemap is ready, you can further customise it using advanced settings. It can add filters to optimum level a sitemap may require.
This is also one of the best sitemap generator tools. It helps you create a deep level of sitemap with pure perfection.
You can use this tool for upto 500 pages for free. However, if you wish to have more than 500 pages in your sitemap, you will have to buy their pro plan. With their pro plan, you get to access their advance sitemap settings like individual category indexing frequency.
This was you can set the frequency for each sitemap. Good part about Screaming frog sitemap generator is that they have easy to use friendly interface and a good support staff.
This is an independent service providing XML sitemap generation. This is a free online sitemap generation service and does not charge anything for the first 500 pages. If you wish to have more than 500 pages in the sitemap, you can upgrade to the pro plan.
It just charges $3.5 per month. Since this is a niche level offering on sitemaps so it really best if you are a pro and want to customize your sitemap to a great level. Generally, big e-commerce websites are using the paid plan of XML-sitemaps.com
This is a free WordPress plugin and can generate excellent sitemaps for your website. This is very user-friendly and even a newbie can understand what configurations to make in it to have better sitemap for the website.
This is also a WordPress plugin with easy UI to handle and take care of sitemap smartly. It has both the versions available. One is free, and the other one is the paid version. Free version will work with 500 and less than 500 links whereas paid version will process unlimited links in the sitemap.
If you are using a self-hosted WordPress blog, I would highly recommend you using Yoast plugin to use for XML Sitemap creation. It not only creates the sitemap but also synchronizes it with the SEO properties of the site.
There are many advanced settings in Yoast plugin that will enable to categorize your sitemaps setting frequency for each category, making proper rules for meta robots and much more. Manual efforts of taking care of such factors get eliminated.
While, if you are a non-WordPress user and have your website made on some other platforms like asp.net or PHP or any other, I would highly recommend you using XML-sitemaps.com. That would solve the purpose to the same level as that of Yoast in WordPress. Your life would be more ease if you get the pro version of the same.
There are many other WordPress plugins that can help you generate a good sitemap. You can find out any and use any of those for generating free sitemap for the site.
I discussed a few crucial points in this article and I would like you to note these take aways from the article :
1. XML Sitemaps are the important SEO factor that can help you place your site better in terms of Search Engine Ranking.
2. You must create a sitemap properly and validate it using various free online validation tools.
3. You must make sure that whatever is blocked using robots.txt and whatever pages that are mentioned in sitemap must be matching. Also, make sure that meta robots also says the same thing as that of xml sitemap.
4. You must always describe the frequency of indexing of pages in your sitemap. This will save the crawl time and not unnecessarily crawl all the pages. Frequency can be used between 0.1 and 1, lowest to highest.