Performance: AI Crawl Control

See which AI crawlers visit your site, how much traffic they bring, and block the ones you do not want.

What it is

AI Crawl Control shows how AI bots (such as the crawlers of ChatGPT, Claude, Perplexity or Google) visit your site, and lets you decide which ones can access it.

To open it: Performance, then AI Crawl Control. It has three pages: Overview, Metrics and Security. At the top you choose the period: Last hour, Last 24 hours, Last 7 days.

If you see "CDN not active", complete the CDN setup first: data is available only when the site runs through Tuurbo. See Performance setup.

Overview

A summary: total AI crawler requests, change compared to the previous period, successful responses, the crawler that downloaded the most content, the most visited page and the visits that AI tools sent to your site.
Metrics: Total requests, Allowed requests, Unsuccessful requests, Total referrals.
Crawlers: one card per company (operator), with its crawlers, allowed requests and referrals.

Metrics

A detailed analysis with charts:

AI crawler requests over time (main crawlers).
Data transferred to AI crawlers.
Response status codes (2xx, 3xx, 4xx, 5xx).
Content format: how many pages were served as Markdown or HTML.
AI referral traffic: which AI services send visitors to you.
Top crawled paths: the pages AI bots read most.

Security

Managed robots.txt: Tuurbo manages your robots.txt file automatically, applying the rules for the AI crawlers you block.
Redirects for AI Training: sends crawlers that collect data to train AI models to an empty static page.
Crawler list: search a crawler, see category, data and requests, and use the Block switch to block or allow it.

Tips

Blocking crawlers used by AI answer engines can reduce your visibility in their answers: block only the ones that collect data for training if you want to stay visible.
Check AI referral traffic to see which AI services actually bring visitors.