Google (and other “legitimate” scrapers) publish the ip ranges they crawl from, anyone claiming to be googlebot (or whatever) but not in the ip range can safely be black holed.

Almost lol: grep, sort and uniq. If I notice someone is hammering my employer's ecommerce site, I'll block them. It isn't required often so I've been reluctant to spend the time setting up fail2ban.

Is it a multi-server setup? If so, do you ssh into each machine and look at the logs?

It's 3 servers so it's not too much hassle to ssh into them and check it manually.

fail2ban