User-agent: YandexBot User-agent: Bytespider User-agent: PetalBot User-agent: TwitterBot User-agent: baiduspider User-agent: DataForSeoBot User-agent: facebookexternalhit User-agent: MJ12bot User-agent: Barkrowler User-agent: Qwantbot # https://support.anthropic.com/es/articles/8896518-anthropic-recopila-datos-de-la-web-y-como-pueden-los-propietarios-de-sitios-bloquear-el-rastreador # User-agent: ClaudeBot Disallow: / User-agent: * Crawl-delay: 20 Disallow: /dl/* Disallow: /download/document/* Disallow: *.pdf$ Disallow: /blog/tag/*/page/* Disallow: /legal Disallow: /privacy # User-agent: AhrefsBot # # https://es.semrush.com/blog/guia-completa-robots-txt/ # User-agent: SemrushBot # User-agent: SemrushBot-SA # User-agent: GPTBot # User-agent: rogerbot # User-agent: dotbot # User-agent: Applebot # # https://commoncrawl.org/faq/ # User-agent: CCBot # # OAI-SearchBot/1.0; +https://openai.com/searchbot # # BrightEdge Crawler/1.0 (crawler@brightedge.com) # # Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot (~500 hits en media hora, ojo) # Java/1.7.0_80 # Apache-HttpClient/4.5.2 (Java/1.8.0_151)