User-agent: * Disallow: /js/ Disallow: /icons/ Disallow: /images/ Disallow: /css/ User-agent: SemrushBot Disallow: / User-agent: AspiegelBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Adsbot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Barkrowler Disallow: /