Quote:
Originally Posted by FitBoop
What does it mean that the bots caught visitors? Will the new VF site be configured to avoid scraping by bots and AI?
|
I was blocking IP addresses that had a lot of hits, so a high probability of either crawling, trying to do a brute force attack, or trying to register a spam account. I did a rough swipe to start because I wanted the CPU to stop maxing out, then went back and refined that list by cross-referencing the IPs I'd blocked to a blacklist IP database. Wendy has a robots.txt file in place, but a lot of these bad entities ignore that directive.
The team feels it's best to move into the vBulletin cloud. That means that vBulletin company will be responsible for keeping the application up to date and performing all the server security and maintenance, including watching out for bots and bad entitites. We'll see what else we can do to block crawlers, but as a public site I don't think there is a foolproof way to stop all of that.