User-agent: Mediapartners-Google* Disallow: # For use by search.w3.org User-agent: W3Crobot/1 Disallow: /webraporu # AltaVista Search User-agent: AltaVista Intranet V2.0 W3C Webreq Disallow: /webraporu # AltaVista Search User-agent: GoogleBot Disallow: /webraporu # exclude some access-controlled areas User-agent: * Disallow: /webraporu