Bot crawlers, also known as web crawlers, are programs that systematically scan and index web pages to collect information from the search engine. It methodically browses the internet and is typically used by search engines to index and rank web pages. However, it applies to a few bots, which is more harm than good, like server load or network instability. You can block these bots with the help of Apache on the WHM/cPanel server.
To Block Bots from Apache
1. Sign in to WHM using your root account.
2. Choose the Service Configuration option from the navigation menu.
3. Click on Apache Configuration.
Configuration Option for the Apache server will be show.
4. Click on Include Editor.
You can apply here for the configurations to your Apache server on a global scale.
5. Visit the Pre Main Include section and, under I wish to edit the Pre Main configuration include file for: Choose All Versions from the dropdown list.
A text box will show in the Pre Main Include section.
6. Copy/enter this code in there and click the Update button.
SetEnvIfNoCase User-Agent "MJ12bot" bad_bots
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bots
SetEnvIfNoCase User-Agent "SemrushBot" bad_bots
SetEnvIfNoCase User-Agent "Baiduspider" bad_bots
Require all granted
Require not env bad_bots
