Are you confused about XML Sitemaps? You might studied a lot but still not clear on how to deal with these?
No problem, today I will talk about Sitemaps in detail and clear all your misconceptions about the same.
Generally, people do not care about Sitemap.When they came to know that is an essential aspect from the SEO point of view, then they are confused on how to proceed further ?
Let me cover all the possible doubts and their solution in this post. I will explain everything in depth and try to quote few examples to give you proper implementation for your usecase.
What is an XML Sitemap?
The XML sitemap is a tree structure of your website that lists down all the important pages that you want Google to consider for your website. Sites without sitemap often notice that their important pages aren’t available in Google index.
There are pages that are relevant and important to the site but are missing any internal link or coverage. Those pages might not included in Google index. But when you especially include those pages in the xml sitemap they are usually indexed.
This tree-like structure or sitemap may differ for different use cases and various sites. It is different for an e commerce website and different for a blog.
Sometimes a site might have more than 1000 pages but only 400 of those are important, then it is must for a webmaster to include those 400 pages in the sitemap.
The sitemap is a tool available with you to tell Google that these 400 pages are the one that you consider are important and should be included in the Google search.
Why do you need an XML Sitemap?
As discussed above, the answer is really simple. You need to have an XML sitemap because you want to tell Google about the important pages of your website and have a preference for them to be listed in Google search index.
Not having sitemap does not mean that your pages will not be indexed. They will be indexed but you do not have any organized approach to find what is important and what is not important.
Let me discuss an example here.
Suppose you are having a blog with 1000 pages but when you perform a Google search, you will probably end up seeing that there are only 600 pages indexed. What about other 400 pages? Why are they not indexed? So it might be due to lesser content or no internal links to those pages and on the top, you don’t have a sitemap that could tell Google that some of these pages may be important.
Ideally, we have two kinds of pages associated with our website :
- Regular Content pages: These are freshly prepared content pages for your site visitor and are meant to be there on search results so that you can drive high-quality traffic to these pages.
If you are having the sitemap, you will be able to categorize between what is important and what is not important.
At first, you would not like to include utility pages in your site map and also mark them with “nofollow, noindex” with meta robots or else block them using robots.txt.
Once you are able to list what are the important pages for your website and found that 300 exact pages are highly relevant on this date and must be shown in Google index.
You can now include these 300 pages in the xml sitemap and leave the rest of the pages. Not including remaining pages in sitemap will not ignore those pages to be indexed by Google but will help the 300 pages to be indexed on priority.
I’m repeating again, by including pages in sitemap we tell Google that these pages are important for the site.
In simple words, we need sitemap to make Google identify pages in the sitemap as important and relevant pages for our website visitors.
I would also suggest using an XML sitemap for your website for the following reasons :
- Improve the crawl times: If you have provided a valid sitemap, it would help search engines like Google to crawl your content faster than to crawl individual pages and then the linked pages separately. Usually the later process is scattered approach and have bad crawl time. However if you have a validated sitemap in your website, you will have much improvement in crawl times.
- Prioritized Content: This is the best way to tell Google about which page has most priority and which page has a lesser priority to you. You can attribute priority anything between 0.1 to 1.
- Media sitemap: For websites that have a lot of images and videos, you can have a separate sitemap for your creative media. This way you help your media content to index faster in search results.
- Performance of Pages: Properly validated sitemap also helps you check the performance of a site as a whole. You will be able to check which page did make to Google SERP and which did not. As you will have your detailed analysis, you will be able to work upon it and further improve the results.
Which pages should be there in the sitemap?
Now that you are clear that we should have important pages in the sitemap. But wait, what’s important?
How do we come to know whether a page is important or not. Now, I would take an example of an e-commerce website so that the relevancy or the importance of a page is clear in terms of including it in the sitemap or not.
An E-commerce website have more than 1000 pages consists of product pages, product categories, subcategories, Seller pages, and utility pages. Now you can define some rules for the sitemap for which pages should be included and which page should not be included.
Not that, if search engine like Google sees the consistency and relevancy of your sitemaps on a regular basis, it starts trusting the website more and thereby increasing the indexing percentage to a higher level.
This categorization of pages can be done dynamically based on specific rules. Although these set of rules will be different for different sites but this example of an e-commerce website, I am defining some rules as examples :
- Product_page.xml: Inclusion of a product page in the sitemap, if it has the product image and the product description more than 400 words. Any product page failing this criterion will not be the part of xml sitemap.
- Category_page.xml: Inclusion of a category page in a sitemap, if the category has more than 5 products and at least 5 of these products are part of product_page.xml sitemap.
This means that any category which has 4 product will not be included in the sitemap. This also implies that if a category has 10 products under it but only 3 of them are the part of product_page.xml sitemap then this category will also not be included in the category_page.xml.
And there can be many such rules depending on what you consider important and relevant for your business.
When we define the sitemap with conditions as listed above, such sitemaps are called dynamic sitemaps. Google usually loves them as they are exact filters to what Google is looking for in a website.
Even for a blog, a sitemap can be split down into 3 to 4 sitemaps.
Creating specific sitemaps with categorization help you to understand which part of sitemap are actually indexed and which are not.
For example: If you submit a sitemap with 500 pages, Webmaster console will tell the number of pages that were indexed. But it will not list down the pages which were indexed and which were not indexed.
So it is a better practice to split these 500 pages into categories and then submit sitemaps. Doing this will enable you to note which category sitemap have what numbers of non-indexed pages.
Having this knowledge will help you figure out why Google did not include them and what possible changes you can make to get them included in the Google index.
Pro Tip : When you place links to utility pages on the homepage of your site, always remember to make them No follow and No index using meta robots. Since these pages are not important and hence transferring the link juice of the home page to these utility pages is just useless.
How to Inform Search Engines About Your Sitemap?
When it comes to informing about your sitemap, you have to make it live on your website in the form of a web link.
For example: example.com/sitemap.xml. After you finish making the sitemap live, you can simply go to the search engine and submit your sitemap.
Every search engine has a method to include the sitemap in its indices. There are many search engines whom you can inform about your sitemap like – Google, Bing, Yandex, Yahoo, and many more.
Let me explain to you about submitting a sitemap to major search engines – Google and Bing.
How to submit Sitemap to Google?
You need to have a Google Webmasters account. Create an account at https://www.google.com/webmasters After you finish the signup process, you simply need to follow these simple steps :
1. Sign in to your Google search console (Webmasters account).
2. You need to select a website in the left sidebar. If you have already added the website there you need to select and if you have not added the website there, you need to first add it in the search console.
3. Now go and click on the “Sitemaps” section in left sidebar to expand it and click on Sitemaps.
4. Now enter the sitemap URL in the sitemap box. Example: https://websitename.com/sitemap.xml
5. Click on Submit button.
Remember, if you have multiple sitemaps then you need to submit all, one at a time. This way you can inform about your website’s sitemaps to Google.
How to Submit Sitemap to Bing?
Just similar to the above process, you first need to have a Bing Webmasters account. So sign up for the one and add your website into it. After you are finish the signup process, you need to follow the simple steps below :
1. Sign in to your Bing Webmasters account.
2. Select the website you just added.
3. In the left sidebar there will be an option for “Indexing Options”, just click over it and then click on sitemap files.
4. Now you just need to enter the sitemap url in the text box. Your sitemap url will look like : https://websitename.com/sitemap.xml
5. Click on Add button. You are finally finish adding the sitemap to Bing Webmasters account.
This is how you informed Bing search engine about your website’s sitemap and it can now understand important pages from your website and can index it.
How to Validate XML Sitemap?
If you do not have a correct sitemap and you are a big brand or you have a good website, you are risking a lot of things on stake. Any site must have a valid and working sitemap in a proper acceptable format.
But how to validate the sitemap? There are various free tools available online that will help you validate your sitemap and if there are any problems with the sitemaps these tools will also help you fix those issues. You can have a look at the following sitemap validators :
There are many other sitemap validators and you can choose the one according to your choice and need. However, all the above-mentioned sitemap validators work well.
Common Validation XML Sitemap Errors:
Although there might be many sitemap validation errors and they can be fixed using above sitemap validation tools.
Some common errors often found in most cases :
1. There are pages which you have already set as “noindex, nofollow” or “noindex, follow”. But by mistake you might have included these pages in your sitemap.xml. This is a case of contradiction. You must think that you are asking Google to index a page by including it in the sitemap and at the same time you are using meta robots to not to index it. This confuses Google and other search engines. The same issue can be resolved.
2. There are few cases where some pages are blocked using robots.txt, but you have included the same pages in the sitemap. This is again a case of contradiction where you might confuse Google on what you want and what you actually ask Google to do.
5 Best XML sitemap generators
So far we have covered almost everything about sitemap and now we actually need to understand how can we generate a valid and perfect sitemap as per our website’s requirement. I will be mentioning here 5 best XML sitemap generator for WordPress that you can wisely use in your WordPress website.
1. Yoast SEO
It’s free. And the best part of Yoast is that it analyses your website thoroughly and then create the sitemap. Once your sitemap is ready, you can further customize it using advanced settings. It can add filters to optimum level a sitemap may require.
If you are using a self-hosted WordPress blog, I would highly recommend you using Yoast plugin for XML Sitemap creation. It not only creates the sitemap but also synchronizes it with the SEO properties of the site.
There are many advanced settings in Yoast plugin that will enable to categorize your sitemaps setting frequency for each category, making proper rules for meta robots and much more. Manual efforts of taking care of such things can be eliminated using the Yoast plugin.
This is also one of the best sitemap generator tools. It helps you create a deep level of sitemap with pure perfection.
You can use this tool for upto 500 pages for free. However, if you wish to have more than 500 pages in your sitemap, you have to buy their pro plan. With their pro plan you are able to access their advance sitemap settings like individual category indexing frequency.
This was you can set the frequency for each sitemap. Good part about Screaming frog sitemap generator is that they have user friendly interface and good support staff.
This is an independent service providing XML sitemap generation. This is a free online sitemap generation service and does not charge anything for the first 500 pages. If you wish to have more than 500 pages in the sitemap, then you can upgrade to the pro plan.
Pro Plan charges $3.5 per month. Since this is a niche level offering on sitemaps so it is really best if you are a pro and want to customize your sitemap to a great level. Generally, big e-commerce websites are using the paid plan of XML-sitemaps.com
This is a free WordPress plugin and can generate excellent sitemaps for your website. This is very user-friendly and even a newbie can understand easily what configurations to make in it to have better sitemap for the website.
This is also a WordPress plugin with easy UI to handle and take care of sitemap smartly. It has both the versions available. One is free, and the other one is the paid version. Free version works with 500 or less than 500 links where as paid version process unlimited links in the sitemap.
If you are a non-WordPress user and have your website on some other platforms like asp.net or PHP or any other, I would highly recommend you using XML-sitemaps.com. That would solve the purpose to the same level as that of Yoast in WordPress. Your life would be more ease if you get the pro version of the same.
There are many other WordPress plugins that can help you generate a good sitemap. You can find out any and use any of those for generating free sitemap for the site.
I discussed a few crucial points in this article and I would like you to note below takeaways from the article :
1. XML Sitemaps are the important SEO factor that can help you place your site better in terms of Search Engine Rankings.
2. You must create a sitemap properly and validate it using various free online validation tools.
3. Make sure the pages that are blocked using robots.txt and the pages which are mentioned in the sitemap must be matching. Also, make sure that meta robots also says the same thing as that of xml sitemap.
4. You must always describe the frequency of indexing of pages in your sitemap. This will save the crawl time and not unnecessarily crawl all the pages. Frequency can be between 0.1 and 1, lowest to highest.