AI
AI Bots Are Overwhelming Small Servers
51% of internet traffic is now non-human, according to Imperva’s 2025 Bad Bot Report. For the Community Science Institute, a small nonprofit running water quality databases for central New York, that statistic became a server problem. A single server took 150,000 page hits over 20 days, roughly 7,000 per day, from machines across the entire world, despite the data being local. Server performance degraded for real users. The culprit was AI crawlers scraping data for model training. The server load was the same as any attack.
Why Standard Defenses Fall Short
Standard tools miss the problem. Throttling is ineffective because most crawlers already observe rate limits, crawling at under 20 pages per minute to avoid being cut off. Even with throttling in place, traffic dropped only 33%. Public IP block lists miss up to 87% of malicious bot IPs because those IPs rotate too fast. GoAccess reports summary statistics that lump human and spider traffic together, its own interface warns “including spiders.” Grep returns individual IPs with no context. Real-time monitors skip historical patterns. The problem requires analyzing logs over time, not reacting to individual requests.
Visualizing Traffic to Separate Machine from Human
Hoetzlein borrowed a visualization from Yungi Kim’s 2018 paper on web server log analysis: plot time on the horizontal axis and host IP on the vertical axis, covering the full IP range of 4 billion addresses. One image shows the entire 20-day log without statistical averaging. Machines appear as regular, repeating dot-dash patterns. Humans appear as irregular, infrequent clusters. Data center attacks show many IPs in a narrow subnet all firing at the same frequency across a class C range. The approach requires no ML model, no pre-training data, just a trained eye and the right plot.
Behavioral Scoring and Subnet Hashing
The scoring algorithm draws on human-computer interaction research. Humans rarely browse one site for more than 6 hours a day, and they don’t access a server for more than 5 consecutive days. More than 400 pages per day on a single server is almost certainly a machine. Four weighted metrics score each IP: request rate, consecutive days, daily range, and daily maximum. Then the algorithm collapses IPs to class C and class B subnets and scores those too, catching data center attacks spread across many addresses. Total reduction: 94%, compared to 33% from throttling alone.
Log RIP: Open-Source Tool and Its Limits
Quanta Sciences released Log RIP on GitHub under Apache 2.0. It takes two inputs: an access log and a config file with the log format and policy thresholds. Output includes raw and filtered traffic PNGs, IP metrics as CSV, and a hierarchical block list for iptables. He walked through the Log RIP build and run demo (27:49), cloning two dependencies and running the full pipeline on a Ruby on Rails log in under two minutes. Two limits: DDoS attacks use randomized IPs that block lists cannot track, and bots that deliberately mimic human timing slip under every threshold.
Q&A
Could geo-fencing by IP and time of day be added to detect bots accessing a regionally focused site at 3am? Hoetzlein confirmed geo-fencing is a planned future feature, noting that open IP geolocation databases already make the data freely available. ▶ 33:28
How frequently do you need to re-run the scan and refresh the block list for a small business? He said it is too early to know, but the plan is to keep collecting incoming traffic continuously even after blocking is active and determine refresh cadence from that data. ▶ 35:17
Notable Quotes
we’re taking a nonAI approach to handling an Rama Hoetzlein · ▶ 3:22
So we found that even when well behaved Rama Hoetzlein · ▶ 16:47
Um we observed a 94% reduction in Rama Hoetzlein · ▶ 15:58
Key Takeaways
- Rate limiting cuts bot traffic by only 33%; behavioral scoring plus subnet hashing cuts it by 94%.
- AI crawlers behave mechanically even when polite — time-vs-IP visualization exposes them without ML models.
- Log RIP is open source, CLI-only, and processes 150,000 log entries in under a minute.