Robots.txt 'Disallow' vs. Meta Tag 'No Index' – What Is the Difference?

If you are an SEO expert or have at least some knowledge of search engine optimization, you are already familiar with the terms “Robots.txt” and “No Index.” If not, the explanation of these terms is quite simple: both Robots.txt “Disallow” and Meta Tag “No Index” are ways of marking web pages that prevent search engines from reading and storing their content in their database, known as the “index.”

Why would you even want to exclude pages from a search engine’s database? Another simple answer: by doing so, we prevent the search engine from prioritizing unimportant pages at the expense of quality ones. Let us take a look at how Robots.txt and “No Index” tags work.

Robots.txt “Disallow”

Robots.txt is a file that you upload directly to your website’s server. You can find it at http://www.YourSite.com/robots.txt. It contains instructions for search engines. When the file has a “Disallow” directive for a specific web page, the search engine recognizes the instruction and knows that it does not need to read that page. With this directive, we are telling the search engine that the web page is not important and can be skipped. This way, we also ensure that pages marked with the “Disallow” directive do not appear in search results.

“Disallow” means “do not read” and “do not look.” The directive does not make web pages invisible but prevents them from being “crawlable.” On a page marked with “Disallow,” incoming links and citations will still be visible to the search engine, which will also recognize them. The search engine therefore knows that the page exists but does not recognize its content. In rare cases, when it cannot offer better search results, it may also offer a page marked with “Disallow,” but the link will be presented without a content description.

As a note, it should be mentioned that smaller search engines do not use Robots.txt files. This means that pages marked with “Disallow” can still be stored in their database, rendering the directive meaningless.

Meta Tag “No Index”

The “No Index” meta tag is a piece of code that you insert into the head section of your web page. Unlike the “Disallow” directive, “No Index” allows search engines to read and recognize the page, but after finishing the visit, they forget that they ever encountered it. The “No Index” directive also applies to all links and citations pointing to a page marked with “No Index.” The page thus remains in no way flagged or present in any database. Additionally, the “No Index” directive is recognized by all search engines.

When to Use Robots.txt “Disallow” and Meta “No Index” Tag

According to some, the “No Index” tag is a safer way to ensure a page is not stored in a database. On the other hand, this method is harder to manage and maintain oversight of, since the directive must be inserted into each page individually. Robots.txt “Disallow” is easier to manage since it consists of just one file.

Every company must first assess its web needs and choose the appropriate directive. You will most likely use the “No Index” directive on pages that absolutely do not need to be stored in a database or that you are creating in secret from your competition. In other cases, Robots.txt “Disallow” will be sufficient.

Do Robots.txt “Disallow” and “No Index” Meta Tag Pass Page Rank?

There is much discussion about whether and how these tags affect and pass Page Rank. If you mark a page with “Disallow” but allow links on it to be followed (“do follow”), the page will still pass Page Rank. However, this Page Rank will essentially go to waste, since the links will not be read by search engines and cannot be forwarded. On the other hand, “No Index” can pass Page Rank if the links on the page are marked as “do follow,” since the search engine reads the page but does not store or index it.