Re: [support] how to stop search engine bots to index specific parts of site

Bots should not be able to get to properly password-protected content.

You can tell bots to avoid specific paths by adding "Disallow: /path/" lines to your robots.txt.

Brian

On Fri, Dec 4, 2009 at 2:45 AM, Armin Medosch armin@easynet.co.uk wrote:

Hi

I am running drupal 5.x and og with panels. We recently started a research group which is protected by password from public, but I think google and other search index bots are still able to index those pages.

What do I need to put into robots.txt to stop the bots indexing this specific og group?

thanks and best regards Armin

-- thenextlayer software, art, politics http://www.thenextlayer.org

-- [ Drupal support list | http://lists.drupal.org/ ]