Mastering Search Engine Control: How to Use NoIndex, NoFollow, and Disallow Directives for SEO Optimization

The noindex directive tells crawlers not to include a webpage in search results.
Indexing is how Google scans (“crawls”) and stores new content for its search index.

There are two methods:

Examples:

⚠️ Note: As of September 2019, Google no longer respects noindex directives in robots.txt. Always use meta tags or HTTP headers.

Google treats NoFollow links as “hints” since March 2020, meaning they may still influence site authority.

Use page-level NoFollow only when:

Mark pages as NoIndex if they offer little value to users or cause duplicate content.

Common NoIndex candidates:

A NoIndex + NoFollow combination:

If a page is already indexed, mark it NoIndex and ensure it’s not blocked in robots.txt, or search engines won’t see the tag.

Example for disallowing one exact URL:

If you own the site, use the Webmaster URL Removal Tool (temporary for ~90 days).
​

For a permanent solution, use NoIndex, robots.txt, or remove the page entirely.

If you don’t own the site, request removal from Google if it:

Generally no — category pages often drive strong organic traffic.
Only consider NoIndex if you have duplicative or auto-generated category pages.

Properly tagged category pages often rank for hundreds of keywords and attract significant organic traffic.

You can block subdomains using:

Password Example: Blocks crawlers and unauthorized users.
Used for:

Robots.txt Example:

For selective blocking, use NoIndex tags on specific pages or directory-level Disallow rules

Accidentally NoIndexing important pages can hurt rankings.

Use Code Injection to add or remove NoIndex tags manually.
Alternatively, password-protect sensitive content.

In Menus & Pages, uncheck “Show this page in search results.”
Note: Hiding a page from menus does not hide it from search.