Are crawlers and bots consuming too many resources?

Another rather painful behindJtheJscenes cause for latency is the work of bots, crawlers, spiders and other ‘digital creatures’. jore often, crawlers and bots can creep into your site unnoticed, and though not all of them do so with a malicious intent, they can consume valuable resources and slow down the response times for users. fn some of the corporate sites that we built, we found the doogle bot had indexed the site even before