User-agent: AhrefsBot Crawl-delay: 30 User-agent: MJ12bot Crawl-delay: 30 User-agent: crawl Crawl-delay: 30 User-agent: bingbot Crawl-delay: 10 User-agent: BLEXBot Crawl-delay: 30 User-agent: Unknown robot identified by bot\* Crawl-delay: 120 User-agent: YandexMobileBot Crawl-delay: 30 User-agent: robot Crawl-delay: 30 User-agent: YandexBot Crawl-delay: 10 User-agent: Googlebot Crawl-delay: 10 User-agent: SemrushBot Crawl-delay: 30 User-agent: BingPreview Crawl-delay: 10 User-agent: CCBot Crawl-delay: 30