AI crawlers are eating your hosting resources

Here is a scenario that did not exist a few years ago and is now routine.

Your bandwidth usage is climbing. CPU spikes at odd hours. Your host emails about resource limits, or your site slows down at times that correspond to nothing in your analytics. Google Analytics shows flat, unremarkable human traffic.

Before you upgrade your plan, check who is actually making the requests. There is a good chance a meaningful share of what you are paying to serve is automated.

Traditional search crawlers were designed to be polite. Googlebot paces itself, respects crawl-delay signals, and backs off when a server slows down — Google has a commercial interest in not knocking over the sites it indexes.

The current generation of AI crawlers behaves differently. They fetch in dense bursts, re-crawl aggressively to keep training data fresh, often ignore caching hints because they want the newest version, and increasingly execute JavaScript — which is far more expensive to serve than plain HTML.

The documented cases are not subtle:

• Read the Docs served roughly 73TB of zipped HTML to one crawler in a year, including nearly 10TB in a single day, producing over $5,000 in bandwidth charges.
• Freelancer.com recorded 3.5 million hits in four hours from a single crawler.
• GPTBot generated around 569 million requests in one month across Vercel’s network.
• Game UI Database measured 200 requests per second and described it as “a two-week-long DDoS attack in the form of a data heist.”
• Wikimedia reported bandwidth up roughly 50%, largely from bots ignoring robots.txt.

Those are large sites. The reason it matters to small ones is that on shared hosting you do not need to be the target.

On a shared plan, dozens of sites draw from one pool of CPU, memory and I/O. If any site on that machine is being aggressively crawled, everyone’s response times degrade. Your site can slow down while your own traffic — human and bot alike — is completely unremarkable.

This is the “noisy neighbour” problem, and automated traffic has made it considerably worse. It also produces a particularly frustrating outcome: hosts enforce fair-use limits, so sustained bot load can get your site throttled for traffic you neither asked for nor benefit from.

Some hosts have started intervening at the infrastructure level. IONOS, for instance, returns HTTP 429 responses to certain training crawlers on shared plans while still allowing user-facing AI search bots through — an explicit trade of some AI visibility for platform stability.

If your speed problems come and go with no pattern you can explain, run the consistency check — variance rather than consistent slowness is the signature of contention.

Compare logs against analytics. This is the definitive test. Google Analytics counts visitors who load and execute JavaScript — roughly, humans. Your server logs count every request. If logs show 400,000 requests and analytics shows 8,000 sessions, most of your server’s work is not going to people.

Most control panels expose raw access logs or an AWStats/GoAccess summary. If you have shell access:

# Top user agents in an Apache/Nginx access log
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

# Just the known AI crawlers
grep -icE "GPTBot|ClaudeBot|CCBot|Bytespider|PerplexityBot|Google-Extended" access.log

Look at the timing. Bot bursts arrive as vertical walls rather than curves — flat, then a spike, then flat. Human traffic has a daily rhythm.

Check what they are hitting. Crawlers frequently target deep archive pages, paginated listings and tag pages that get almost no human visits. Heavy traffic to page 47 of your category archive is not an audience.

Work through these in order. The early ones are free.

Compliant crawlers respect it. This is a request, not enforcement, but it handles the well-behaved majority:

# Training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Slow the rest down rather than blocking
User-agent: *
Crawl-delay: 10

For crawlers that ignore robots.txt, you need something with teeth. Cloudflare’s free tier includes a one-click toggle to block known AI scrapers, plus WAF rules matching on user agent. This is the highest-leverage step for most people because it stops the load before it reaches your server at all.

Crawlers hammer the same high-value pages repeatedly. Full-page caching means those requests get served without touching PHP or your database, which converts an expensive request into a cheap one. This helps with bots that you have decided to allow.

If you control the server, cap requests per IP per second. Nginx’s limit_req, Apache’s mod_evasive, or fail2ban all do this. Rate limiting is gentler than blocking: it slows crawlers down rather than cutting them off, which is often the right balance.

If genuine human traffic still needs headroom after you have filtered the noise, upgrading is reasonable. Doing it first means paying for capacity to serve requests that will never buy anything from you.

Most advice on this topic is written by hosting companies selling dedicated servers, and it skips the part where blocking has a cost.

There are two categories of AI bot, and they are not equivalent:

Training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended — collect content to train models. Blocking them costs you nothing in traffic. Your content simply is not included in the next model.

Retrieval bots — OAI-SearchBot, ChatGPT-User, PerplexityBot and similar — fetch pages because a user asked a question right now, and they cite sources. Blocking these removes you from AI-assisted answers, which is a growing referral channel.

Blanket-blocking everything with an “AI” label throws away the second category to solve a problem caused by the first. The defensible position for most sites is: limit the training crawlers, allow the retrieval bots, and cache well enough that being cited costs you almost nothing.

This is partly a hosting-choice problem, and it is starting to differentiate providers.

Bot management is becoming a baseline platform feature rather than a premium add-on. Some managed hosts now filter aggressive crawlers at the edge automatically and exclude suspected bot traffic from billable metrics. On a plain shared plan you typically get neither, and you are exposed to every other site on the machine.

If you are being throttled for load you did not create, the honest options are an isolated environment or a host that handles this for you. Our speed rankings cover eight managed cloud providers, and the comparisons work through the differences in what each platform includes.

Before you spend anything, though, read your logs. The cheapest fix in this entire guide is a robots.txt file and a free Cloudflare account — and for a lot of sites, that is the whole solution.