Why And How To Discourage Search Engines From Indexing WordPress Content

Yashwant Shakyawal SEO Leave a comment 29 Views

In this article, we will discuss why and how to discourage search engines from indexing WordPress content. So keep reading.

As business owners, we always dream of having as much traffic coming to our WordPress site as possible. More visitors equivalent to a higher rate of leads and potential customers.

To appear on search results and receive traffic, your site firstly has to get crawled and indexed. While many people scramble to improve SEO and websites’ search engine visibility, there are still some cases when you don’t want crawlers to find their sites.

For instance, you probably launch a test website for development purposes or simply provide restricted content for certain users.

Feel curious? Read on to discover other scenarios of deindexing your site, including both content, and media files. We’ll then show you 6 ways to discourage search engines from indexing your site. These methods will vary in complexity, from the website’s built-in feature to using a supporting tool.

3 Main Reasons Why You Shouldn’t Allow Search Engine Indexing
Use WordPress Default Visibility Option
Edit Robots.txt File Manually
Use “noindex” Meta Tag
Password Protect Your Entire Site
Block Search Indexing Your Files
Remove Indexed Content from Google

3 Main Reasons Why You Shouldn’t Allow Search Engine Indexing

You might not notice but there are numerous causes to shield your content from search crawlers. The 3 situations below will quickly explain the purposes of your uncommon action of removing search indexing:

1. Hide Duplicate Content

Duplicate content presents as one of the main SEO issues on your site. Google will get confused and possibly give a penalty on your content when finding similar pages and posts.

Instead of ranking both, this giant search engine will try to figure out which one is the original. Your content will be likely removed from the top pages of search results which hugely impacts your traffic.

If there is a demand to publish duplicate content, you must block Google bots from indexing them. Take examples of an advertising landing page or a staging site. After the promotion period ends, these pages are not designed for the public though. So don’t let it get indexed and listed in SERPs.

2. Restrict Content Access

You provide some kinds of private content for specific users such as members or paying customers. As only certain users can gain access to these pages and posts, they should not be available on search result pages.

Intranet websites for internal project management is another case that should be taken into account. You build a site to share the project progress with clients, train newbies, or assign tasks to team members. You must make sure to keep prying eyes off this informational commercial property.

3. Put Your Site Under Development

While preparing your website for launch, it’s necessary to hide your site under development. You don’t want visitors to see the incomplete work which might leave a bad impression and affect their experience. They might think the site is broken and the chance that they come back to your site is very low.

The same thing goes for test websites that are used to check your new features or new page designs. Since it’s not indexed or ranked anywhere on search results, nobody but only you can see it. You’re able to see how they live page looks to make adjustments easily without influencing the user experience.

Once understanding the reasons to hide your site from search crawlers, it’s time to take action! There are multiple methods for you to apply, from manual to automatic options.

1. Use WordPress Default Visibility Option

If you’re trying to look for a simple yet effective solution, this default visibility option takes that weight off your mind. All you need to do include:

1. Log into your WordPress admin dashboard
2. Head to Settings on the left-hand navigation menu

1. Scroll down and tick the box of “Search engine visibility”

Remember to save your changes

That’s it! By enabling this feature, WordPress basically tells all search engines (Google, Bing, or BuzzSumo) not to crawl your site. However, if you don’t have serious issues requiring you to shut the site down immediately, it’s not recommended to use this feature.

Why You Should Rarely Check Search Engine Visibility Box

Once checking the “Search Engine Visibility”, WordPress edits the robots.txt file and at the same time, it adds a line to your website header.

As we can see, the code forces crawlers not to index and follow your content. In fact, some search engines ignore this request and still index some of your pages, posts, and media files in particular.

Plus, your SEO will be destroyed if you use this method for a long time. Right after making the site invisible from search engines, you might lose all its rankings. Your SEO effort put into the site becomes meaningless now.

Another situation is that after completing adjustments on your site, you have to make it live again. You somehow forget to uncheck the box. Consequently, the entire site never shows up on search results again.

2. Edit Robots.txt File Manually

You’re techy and prefer editing the robots.txt file manually? This method is for you. To get started, you need access to your File Manager or an FTP client. The guide below will show you how:

Log into cPanel and go to Files → File Manager
Under the public_html, find the robots.txt file. In case you don’t see it, simply create a new one
Right-click the file to edit it
Fill in the following syntax there

User-agent: *

Disallow: /

If you plan to discourage search engines from indexing specific pages, just add the slug of that page after the “Disallow:/” rule. For example: Disallow:/blog/improve-your-content-seo.

3. Use “noindex” Meta Tag

A “noindex” meta tag permits you to prevent a page from appearing in Google search. You can add it to the page’s HTML code or return the “noindex” header in the HTTP request.

Please include the following code in the <head> section of your desired page.

The above tag blocks all kinds of search engines from indexing your site. To narrow down to certain ones, replace “robots” with a specific crawler such as:

4. Password Protect Your Entire Site

Another way to prevent search indexing your site is password protecting it. Most password protection tools enable you to lock your WordPress pages and post with passwords and at the same time, discourage search engines indexing.

Trusted by over 10 thousand users, Password Protect WordPress (PPWP) Pro plugin authorizes you to shield the whole WordPress site under a password form. Once protected, your content won’t appear anywhere on search results by default.

To get started using the plugin, you need to

Download and install the PPWP Pro plugin, which is wrapped in a zip file format
Log into your WordPress admin dashboard
Go to Plugins → Add New → Upload Plugin then choose the zip file you’ve just downloaded
Activate the plugin
Visit its Sitewide Protection page right on your WordPress navigation menu
Enable the option Password Protect Entire Site and enter your passwords
Save your changes

The plugin can handle up to 10 thousand passwords, depending on your demands. Visitors find no way to access your website now unless they have the correct key. Out of the box, it gives you the right to exclude specific pages from the site-wide protection status.

You need to switch on the Exclusion option and enter the pages you want to exclude there. As a result, they will be still accessible to all users.

On top of that, you can use this plugin to protect individual pages and posts with passwords too.

As a matter of fact, this password protection method doesn’t work for media files. Although your pages and posts are covered under a password form, their WordPress blog images or video keeps displaying on Google image search. To completely discourage search engines from indexing, you need help from Prevent Direct Access (PDA) Gold plugin.

5. Block Search Indexing Your Files

Prevent Direct Access (PDA) Gold permits you to protect all types of media files on your WordPress site from unwanted users. It tells Google and other search engines to stop indexing any of your protected files. And the outcome is that your secured files and their original URLs won’t appear anywhere on search results.

Discover how to use the plugin via the following guide:

Install and activate the PDA Gold plugin. You can refer to the guide for installing the PPWP Pro plugin above.
Head to the bottom of the plugin Settings page to make sure the feature Grant Web Crawlers Access is switched off
Go to the Media Library section in your WordPress admin area
Find the files you need to protect
Click the Configure file protection link on the right side of your files

6. Remove Indexed Content from Google

Are you afraid of pages already being indexed? Don’t worry! Keep reading to learn how to remove them from SERPs:

Access your site Google Search Console. Please set up your Google Search Console account if you don’t have one.
Move to Legacy tools and reports and click Removals
Click the button of Temporarily hide then enter your desired URL
Choose Clear URL from the cache and temporarily remove it from the search before Submit Request

Now, Google will temporarily remove your page from search results. Instead of filling a specific page URL, you can enter the site’s domain to discourage search engines from indexing the entire site.

Ready to Discourage Search Engines from Indexing Your Site?

You have numerous reasons to discourage search engines from indexing and showing your site on SERPs. Maybe you want to hide duplicate content or put your site under development so that no one can see the uncompleted work. Some site owners also don’t like to see their restricted content to be visible by the general public.

Fortunately, there are multiple ways to block search indexing for you to choose from, for both non-techies and code-savvy. WordPress default visibility feature gives you the right to take action with only a mouse click. However, this method is not as effective as we think since it might damage our site SEO.

Editing the Robots.txt file manually or using the “noindex” meta tag requires more effort. You have to enter code in your File Manager or at the head of the page.

PPWP Pro, on the other hand, will wow you with the ability to discourage search engines from indexing your password-protected content. Still, it doesn’t support media files. Then, you need help from the PDA Gold plugin. It stands out to be an experienced veteran in securing WordPress images as well as removing them from search results.

If you have any questions related to your website’s search engine visibility, feel free to post them in the comment section below! We’ll sort them out as soon as possible!