Thursday, November 24, 2005
Maybe you don't want every blasted search engine in the universe beating a path to your site and listing every page. You can communicate to crawlers your intent to exclude a specific page from indexing by placing a META tag within the page's document head. The name of the META tag in this case is robots, and the possible instructions are index (or noindex) and follow (or nofollow). Index means that robots record the page's contents for use in a database or search engine. Follow means that the robot follows all the links on that page to collect more data.
To invite a robot to index and follow a page, the tag would look like this:
To instruct a robot not to index or follow the links on a page, the tag would look like
To invite a robot to index and follow a page, the tag would look like this:
To instruct a robot not to index or follow the links on a page, the tag would look like
