User-agent: * Disallow: /ajax/ Disallow: /set-lang/ Disallow: /en/set-lang/ Disallow: /en/ajax/ Disallow: /fr/set-lang/ Disallow: /fr/ajax/ Disallow: /de/set-lang/ Disallow: /de/ajax/ Disallow: /es/set-lang/ Disallow: /es/ajax/ Disallow: /it/set-lang/ Disallow: /it/ajax/ Disallow: /short/ Sitemap: https://s3.amazonaws.com/watchpatrol-storage/sitemaps/sitemap.xml User-agent: Yandex Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: grapeshot Disallow: / User-agent: proximic Disallow: / User-agent: BLEXBot Disallow: / User-agent: Baiduspider Disallow: /