Yeah, of course if you're a large service, stuff that works for SMEs isn't gonna work for you...
If your problem is AI crawlers, then simple rate limits help, I've helped countless of businesses with this already. For the ones that it isn't enough, you continue adding more roadblocks. There is no "one size fits all here" and that you seemingly is under that belief, leads less credence to what you're saying, not more.
Brother if you have the solution, start a company.
There are people willing to throw money at you.
But Free is hard to beat with a global presence.
And please listen to what others are saying here, there's a lot of experienced people here. It is no longer 1998, 2008, or even 2018. The traffic a basic website experiences now is a massive increase, especially for those well SEO'd and using TLS.
Also you said this:
> CDN is something you do once you run out of options, not something you should reach for immediately, it makes no sense in most cases of just hosting a website.
Origin obfuscation.
Also greatly helps to protect it directly from various attacks and scans, especially in our new LLM driven security world where new 0days are being found constantly. How much at risk are you willing to put the SMB?
If your problem is AI crawlers, then simple rate limits help, I've helped countless of businesses with this already. For the ones that it isn't enough, you continue adding more roadblocks. There is no "one size fits all here" and that you seemingly is under that belief, leads less credence to what you're saying, not more.