[support] how to stop search engine bots to index specific parts of site

[Brian Choc]

Bots should not be able to get to properly password-protected content.

You can tell bots to avoid specific paths by adding "Disallow: /path/" lines
to your robots.txt.

Brian



On Fri, Dec 4, 2009 at 2:45 AM, Armin Medosch <armin at easynet.co.uk> wrote:

> Hi
>
> I am running drupal 5.x and og with panels. We recently started a
> research group which is protected by password from public, but I think
> google and other search index bots are still able to index those pages.
>
> What do I need to put into robots.txt to stop the bots indexing this
> specific og group?
>
> thanks and best regards
> Armin
>
>
> --
> thenextlayer software, art, politics http://www.thenextlayer.org
>
>
> --
> [ Drupal support list | http://lists.drupal.org/ ]
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.drupal.org/pipermail/support/attachments/20091204/444b2dbf/attachment.html