How to Stop Search Engines from Indexing a Page

If you have a website, you’ve probably wondered at least once how to stop search engines from indexing a page. 

There are numerous scenarios when such a move would make sense, and there are multiple ways to go about it. 

In this guide, we’ll show you how to prevent search engines from crawling one or more pages on your site. 

Why would you want to stop search engine indexing on a page?

The first question that may pop into your mind when reading the title of this article may very well be, “Why would I want to prevent search engines from indexing my site?”

And you’d be totally right to be puzzled by the idea of blocking search engines. After all, the whole point of building a website is to rank in search engine searches, not be ignored by them. We even go to the trouble of updating existing content in order to alert search engines to check out and index our pages regularly to improve our search engine optimisation.

However, there are multiple situations when turning off search engine indexing for certain pages makes sense. 

Maybe you’re still developing your website on a domain name that’s publicly accessible, and you don’t want search engine indexing turned on while your site is in maintenance mode.

Or perhaps you’re creating a private blog that you don’t want to be accessible to the general public so you don't want it popping up on search engines. 

Another reason you might want to do this is to ensure that outdated, thin, or redundant content doesn't get included in the search engine index, potentially leading to a poor user experience.

Finally, many people use WordPress sites for intranets or project management and want to avoid making internal documents accessible to users outside their organization, so they turn off search engine indexing.

What web pages don’t need to be indexed?

Besides the reasons listed above, you may also want to stop search engine indexing on various pages on a live, public website. 

Many blog owners don’t know this, but not all pages on a website need to be included in search engine indexing.

It is essential to manage search engines crawling certain pages to optimize your website's performance and relevance.

Some of the pages that you should de-index include the following:

Admin and login pages

You shouldn’t have your admin and login pages appearing in Google search results so it's a good idea to turn search engine indexing off. To prevent the pages from being indexed, you can simply add a no-index tag to them. The only exceptions should be those login pages that serve a community.

Thank you pages

These are those pages you use to thank your newsletter subscribers or new customers/commenters. They are considered thin content pages, and since they are of no value to search engines, you shouldn’t have search engine indexing turned on for these.

Author archives if you’re the only author on your blog 

Google and other search engines may consider author pages duplicate content if there’s only one author on the entire website. 

You can either disable the author archive to prevent that from happening, but if, for any reason you decide to keep it, you should no-index your author archives. 

Internal search results pages

An internal search results page is a web page that results when a user performs a search on your site using the search function. 

You shouldn’t have any of your internal search results pages indexed by search engines because they are likely to harm your user experience and your website in general.

Google’s Quality Guidelines specify that you should prevent crawling to search result pages using robots.txt because they don’t add any value for users who come to your website from a search engine.

How to stop search engines from indexing a page?

There are multiple ways to discourage search engines from indexing a page. The easiest of them is to include the no-index tag, but if you can’t do that for some reason, there are other ways to de-index a page.

Here’s a quick overview of the most common methods.

Add a “noindex” meta tag

Adding a “no-index” meta tag to a page sends a signal to search engines that it cannot add it to the search index, even though it can still crawl it. 

In other words, if you add a noindex meta tag to any page on your website, it won’t be shown in the search engine results.

The good news is that it’s quite easy to use metatag to prevent a specific page on your website from appearing in SERPs. 

Here’s how to do it.

To do so, go to the beginning of your header by opening the source code of the page you want to de-index and paste the tag into a new line within the header.

Once you’ve added the tag, the search engine won’t index the page, and it won’t appear in any SERPs.

It’s important to note that you need to do this for every page that you want to de-index.

If you also want search engines to not follow the links on the de-indexed page, you should use the “noindex,nofollow” metatag instead: 

< META NAME =”robots” CONTENT =”noindex,nofollow”>.

Add a meta robots tag

The purpose of a robots.txt file is to tell search engine crawlers which pages they can access on a site.

According to Google, you can use a robots meta tag if you want to control how a specific page should be indexed.  

This means that you can hide certain media files from search engine crawler bots, such as images and videos, to stop them from appearing in search results. 

Setting up your robots.txt file is pretty technical, so it’s a good idea to consult Google’s developer documentation. 

Remove the URL from Google Search Console

Removing the URL from Google Search Console only works for Google, but it’s a quick way to stop the most popular search engine indexing a page on your site. 

To do so, you can use Google Webmaster’s Remove URL Tool within Google Search Console. According to Google, you should employ this method when you need to take off a specific URL from Google Search quickly.

The URL in question needs to be on a Search Console property that you own. This method will only temporarily block the URL.

Here’s how to use the Google Remove URL tool to exclude pages from search engine results in Google Search Console.

Completing this action will clear the page's cached copy from Google’s index and hide it from Google search results. 

In Google Search Console, you can check out the Index Coverage report. It shows which pages Google has indexed and alerts you to any issues that need attention. If you've noticed that some pages should not be included in the search index, this is an excellent starting point.

Managing your XML sitemap is key for better site indexing. Think of an XML sitemap as a roadmap for search engines, helping them find and index all the important pages on your site.

By submitting an updated XML sitemap to Google Search Console, you can let search engines know which pages to focus on and which ones to ignore, making your site's content more relevant and focused.

Use a plugin to noindex one or more pages in Google

Another easy way to noindex a single page or multiple ones in Google is with the help of a plugin.

This is the perfect solution for beginner users who don’t have a lot of coding experience and would like to be able to noindex pages in Google with just a couple of clicks.

There are multiple plugins that make it easy to noindex pages in Google, with Yoast SEO, Rank Math, and AIOSEO being the most popular.

Let’s have a look at how you can noindex pages using each of these plugins.

Yoast SEO

If you’re running Yoast SEO, all you have to do to noindex a page is change a simple setting.

To turn off website indexing, go to the Advanced tab in the Yoast SEO meta box and select “No” for the question “Allow search engines to show this post in search results?

The default setting for this box should typically be “Yes,” and it’s the setting that you’ve probably set when installing Yoast.

To noindex posts, simply select “No” manually for every page or post that you don’t want to appear in any search engine's index.

If the post you’re deindexing has already been crawled by Google and appears in the SERPs, it will take a while for it to disappear after turning website indexing off.

Yoast SEO assumes that you want to set your posts to “no index, follow,” which means that Google and other search engines will continue to follow the links on those posts.

If you don’t want that to happen, answer “No” to the next question, “Should search engines follow links to this post?”

Rank Math

If you use Rank Math instead of Yoast SEO, it’s still very easy to noindex one or more pages on your site.

Just like in the case of Yoast SEO, Rank Math indexes your posts by default. If you don’t want a particular post or page to be indexed, you need to set them to “No Index” manually.

Go to “Advanced Mode” and check the “No Index” check box that’s located in the Robots Meta section.

Alternatively, you can set a page to noindex from the admin menu bar. Go to Rank Math SEO > Mark this page > As NoIndex to do so with just one click.

AIOSEO

As you might expect by now, the default setting in AIOSEO is to index all the pages and posts.

To exclude specific pages, you need to do so manually for each post or for entire categories.

To do so, scroll down to AIOSEO Settings and go to the “Advanced” tab. In the “Robots Setting” category, you’ll see the “Use Default Settings” setting turned on.

Change the toggle to off to reveal the Robots Meta heading. Check the No Index checkbox (and the No Follow one if needed), then click “Update” to apply the new settings to your post.

Password protect your entire site via cPanel

Using a password to protect your site via cPanel is another way to prevent search engine indexing for a site. 

To access a password-protected site and see its contents, users will have to enter a username and password.

You can protect your entire site via cPanel by logging into your hosting account and navigating to “Password Protect Directories”. The location of the icon may differ from one host to another, but you can generally find it under the “Security” section. 

Choose the public_html directory or the folder where you have installed WordPress if it’s a different one.

You’ll then have to set permissions for the folder and create a username and password.

This will add password protection to your entire site, which means that whenever search engine crawlers or users attempt to visit the site, they’ll be prompted to enter the username and password.

Password protect your WordPress site with a plugin

If you don’t have access to cPanel via your hosting or you don’t want to go down that route, you can use a plugin to password protect your site.

Some popular plugins for password protection include: 

Password Protected 

This plugin is one of the easiest ways to password protect your site. You only need to check a box to enable protection, and there’s no need to create a username.

One of the best things about the Password Protected plugin is that it offers the option to whitelist IP addresses as well as various types of requests. 

You can easily configure these options by going to Settings → Password Protected after you install and activate the plugin.

SeedProd 

Another plugin you can use to password-protect your WordPress site, SeedProd is a popular plugin that also comes with access control and permissions features. 

By adding a username and password in SeedProd, you can hide your website from users and search engines alike. 

Passster

With Passster, you can password protect your entire site or specific pages or posts. Besides restricting content with a password, this plugin allows you to customize the headline and restrict content with a captcha.

Other options include multiple passwords to protect your content and unlocking protected areas of the website via a link. 

Conclusion

As you can see, there are multiple options available if you want to stop search engine indexing for a page. 

Whether you’re adding a “noindex” metatag or using a plugin to password protect your entire site, it’s quite easy to prevent search engines from indexing one or more of your web pages.

Hopefully, our guide has made it easier to choose the best option for your particular website and type of content.

Related Posts

Promised An SEO Guarantee? Here’s What You Need To Know

What is the Average SEO Cost? Tips to Plan Your Spending

13 Best Websites to Help You Get High-Quality SEO Leads

Website Architecture: 9 Proven Tips to Boost Your SEO Success

How to Find Expired Domains and Boost Your Website’s Success