Search

How To Check Which Pages Are Indexed By Google? + [Why They Aren’t Being Indexed and How to Fix It]

By Peter Karanja |

Knowing how to check which pages are indexed by Google is a necessary process when conducting a site audit.

Indexing is Google's way of “allowing” you to appear on search results, or even on Google Discover results. If your pages aren't indexed, you won't appear, and you will not get organic traffic.

However, not all pages will get indexed, and you won’t realize it until you check.

And in this article, I'll show you how and give you some tips on how to get pages indexed more easily.

How To Check Which Pages Are Indexed By Google?

Below are three ways to know which pages are indexed.

Google Search Console

There are two ways to check which pages are on the Google Index using Google Search Console;

Using the URL Inspection

If you have a few articles on your site or recently published some articles and want to check if they're indexed, you can use the URL Inspection as shown below.

How To Check Which Pages Are Indexed By Google?

All you have to do is enter the page's URL in the search box and click enter. If the page is indexed, you’ll get the following notification.

How To Check Which Pages Are Indexed By Google

If it’s not indexed, you’ll get the following notification.

How To Check Which Pages Are Indexed By Google?

As you can see above, the page has been discovered but not indexed. I had just published that article. That's why it was not indexed. You can wait for a few days for Google to automatically index the page or request indexing. But there's no telling how long it will take.

Using the Coverage Feature

On the left-hand side of Google Console, you'll see the Index section, which has Coverage, Sitemaps, and Removals. Click on Coverage, and it will show the total number of indexed pages labeled as valid.

Pages that are not indexed are labeled as Excluded. And you’ll also see the pages with errors. Below is an example of how the page looks like;

The good thing about the Coverage feature is that it tells you why exactly the pages aren’t indexed. For instance, you could get a response saying that the URL is not on Google and a Sitemap: N/A tag on the Coverage, which means the page can be indexed, but it's not on the XML sitemap.

Robot.txt and canonical issues are also shown in the Coverage section, but I’ll discuss what to do about them later in the post.

Google Site Query

This is another easy way to check the indexed pages. Go to Google Search and enter site:yourdomain.com. For instance, if your website is ilovegardening.com, you will enter sitse:ilovegardening.com, and you'll get a complete list of all the indexed pages. Below is how the page would look like;

How To Check Which Pages Are Indexed By Google?

You can also narrow the search by adding more parameters to the command. Below are some of them;

Using Google Indexed Pages Checkers

There are several tools that can help you find the indexed pages more efficiently. These tools include;

The Crawling Process

When you search a particular query on Google, it takes less than a second to get the results. But a lot happens in the background, which can help a website owner increase their chances of being indexed and appearing on the search results. To understand how it all works, let's first define some of the most common terms used;

How It Works

So, when a user enters a query on Google, the Google bot crawls all the pages in the index and uses the links on those pages to find more relevant pages. This process goes on until there are billions of pages. Google then filters these pages by querying them based on relevance to give you the best answer to your question.

Some of the parameters the Googlebot checks include the page rank, the quality of the website, keyword placement, and the number of websites linking to that page.

The result will be all the sites that answer the user's query, the URL to the page, and a short snippet that will help them decide which page they should pick.

They also get several related searches they can try, and you, as the website owner, can use these related searches to find more queries your audience may be interested in.

Google’s crawling process is free and mostly automatic. You are not required to pay anything to get your site crawled or ranked. All you have to do is submit your site’s XML sitemap and make sure your site follows Google’s webmaster guidelines.

As I mentioned earlier, you can also request crawling by requesting indexing. Sure, you might find ads ranking above other content, but even the ads are shown based on relevance and not how much the advertiser paid.

Why Isn’t Google Indexing Your Site

Now that we know how indexing works, let’s find out why your pages may not be indexed.

You’ve Not Submitted Your Sitemap

As mentioned earlier, you must submit your sitemap.xml file for Google to crawl your website. It’s easy to forget about this process, especially with your first website. Or you may have submitted it, but there was an error, and you didn't realize it.

Either way, if Google is not indexing your pages months after you created them, the first thing should be to check and resubmit your site’s Sitemap, which is easy to do, especially for WordPress websites. All you need to do is install the Yoast SEO plugin, which automatically creates the sitemap file.

Just to be sure, go to SEO –General –Features, and make sure the XML Sitemaps is on. Then, type this: https://yourdomain.com/sitemap_index.xml or https://yourdomain.com/sitemap.xml in the Google Search box, and you should see your site's Sitemap. Then, copy this URL and submit it in Google Search Console – Sitemaps. And that's it.

Besides submitting your Sitemap, you also need to ensure all pages are indexed. While Google can easily find all the pages that need to be indexed, you can make it even easier by adding any pages that aren't on the Sitemap.

Crawling Errors

If your pages aren’t being indexed, there could be crawling errors. The Google crawler often crawls a site to check if the pages have changed or if there is new content published. But this is not always successful due to DNS errors, server errors, URL errors, robot.txt, and many other errors.

You can inspect your website’s crawl health by going to Settings – Crawl Stats, as shown below.

Here, you will get the total number of Crawls within a certain period, the average response time, and the total download size. You can then crosscheck the results on that page with the documentation on this page to find a way to fix them.

Robot.txt or Noindex Tags Blocking Some Pages

One of the Crawl errors you may be getting is a Robot.txt file blocking some pages. In most cases, the blocking is done intentionally when you want to instruct Google’s Crawlers not to index a page. But there are instances where it's blocking pages you need to be indexed using the ‘noindex' tag or the following code snippets.

1 User-agent: Googlebot
2 Disallow: /
 
1 User-agent: *
2 Disallow: /

Noindex tags also appear in your site's meta tags. Therefore, you need a thorough website audit to make sure the pages you need indexed aren't blocked.

You could get this information from the Coverage section of GSC, using a site audit tool like ahrefs, or manually checking your site's code and the robot.txt file. And removing them isn't that complicated either. Once you find these tags, just manually delete them from the file, and Google will start crawling those pages.

Duplicate Content

If several pages on your site return similar content during crawling, your pages may not be indexed. Some of the reasons you have duplicate pages include;

You can eliminate duplicate content through the following ways;

Site Speed

According to this study, a site's loading speed affects how often Googlebot crawls your site. You would want Google to crawl your site often if you constantly update your site's content.

But if a page takes more than 3 seconds to load or doesn't load at all, Googlebot will have a problem indexing it.

Also, if pages are taking too long to load, the Google Crawler may index only a few pages on your site while you need it to crawl and index more pages. Site Speed is also known to affect overall rankings, where slow sites rank lower than fast sites. Google even released a core update based on Core Web Vitals in June 2021.

Privacy Settings and .htaccess Files

For those with WordPress sites, altering the Privacy settings can also affect crawling and indexing. To rule out this issue, login into your WordPress admin panel, then go to Settings – Privacy and make sure it’s off.

On the other hand, .htaccess files help with SSI, Mod_Rewrites, Hotlink Protection, Browser Caching, and other functions on the server. But this file may also affect the site’s speed and interfere with the crawling process.

Your Site Was Penalized

If you don’t follow Google Webmaster’s guidelines, your site will lose earnings, traffic, or worse, your pages may be removed from the Google Index. As long as your site follows the Google Webmaster Guidelines, you don’t have to worry about penalties. And you’ll always get a warning before they take such drastic action.

How to Get Your Pages Indexed By Google

I've highlighted several times that you can get your pages indexed quickly by requesting indexing. This is by submitting the URL to the URL Inspection tool to check its status, and you'll get an option to request indexing. Other methods I’ve mentioned include;

 If you did this and your pages are still taking too long to be indexed, below are other tips you can try.

Internal Linking

As mentioned earlier, the crawling process involves following links that are on the page. So, if there are “orphaned” pages or pages that you've not linked to, Google bots may have a hard time finding and indexing these pages.

Conducting a site audit with tools like Ahrefs or SEMrush can help you identify the orphaned content. But if you want to get deeper insights into the whole internal linking process and prevent orphaned content, you should try Link Whisper.

This is a tool designed by Spencer Haws that helps suggest articles to link to based on relevance. It also helps if you start by linking from your best-performing pages – the pages that Google crawls often.

Besides ensuring healthy internal linking, you should also ensure the internal links are not nofollows. Why? Google bots don't crawl nofollow links. You could assign a nofollow tag to outbound links when you don't want to transfer PageRank to another site, but don't do that for internal links.

Publish High-Quality Content and Remove low-Quality Pages

Indexing is not all about technical stuff. You also need to publish content that's valuable and helpful to the reader.

Google bots will first scan the page to determine if it serves the user's intent before scanning the links on that page. If it's a low-quality page, it may not be indexed, and other pages that page links to may not be crawled. As mentioned earlier, Google has a crawl budget.

If you have several low–quality pages on your site, you may deplete the Crawl budget before the high-quality and relevant pages are crawled. But this primarily applies to sites with thousands of pages.

Besides removing low-quality pages, you can also improve the site’s speed, remove duplicate content, fix or remove pages with errors, and enhance internal linking (I’ve discussed how you can do all this in the previous sections).

Other aspects to pay attention to include the headers, tags, and the sites you link to.

Backlinks

Sites that link to your site also matter. Backlinks show Google that your site has more value, and Googlebot will likely crawl them more often than those without.

Share Your Content on Social Media Channels

Sharing your content on social media is an excellent way to get your business out there, improve rankings, generate traffic and leads.

But what you might not know is sharing your content also creates social signals, which can help trigger Google Crawlers to crawl and index your pages. Some of the places you can publish your content to speed up indexing include;

How to Check Which Pages Are Indexed By Google

And there you have it. If your traffic isn't growing as you'd expect, or you've experienced a drop, chances are Google crawlers are having trouble crawling and indexing your pages.

I've detailed all the different ways you can check which pages are indexed by Google, identify indexing issues, and fix them.

Remember, not appearing in Google Search results doesn’t mean you’re not indexed. You may be indexed but are ranking very low since your site lacks authority.




Blogging & Niche Websites

By Peter Karanja

Peter Karanja is a seasoned freelance writer with lots of experience in helping clients produce content for their niche sites.

He spends most of his time learning how to create better content.

When not working for clients, he spends most some of his time building his niche sites.

Want to learn step-by-step how I built my Niche Site Empire up to a full-time income?

Yes! I Love to Learn

You may also like

View all

Recent Posts

View all