Blocking Bad Bots and Scrapers with .htaccess

Why block "bad bots"? Well, first let me say what a "bad bot" is. It is a search engine spider or email harvester spider that downloads pages from your website and does not obey your robots.txt file. The major of bad bots are a useless waste of your server's bandwidth and resources. That's why I ban them from my websites.

Most bad bots can be identified by their User-Agent value. By making your server check for known User-Agent values of bad bots, you can block those bad bots. Some bad bots forge their User-Agent value and pretend to be web browsers such as FireFox or IE; those bad bots cannot be blocked based on User-Agent (you may be able to block them based on IP address).

The testing for bad bots goes in your website's .htaccess file. If you have a dedicated server, you could put the testing in your webserver's http.conf file for greater efficiency.

I have the following statements in my website's root .htaccess file before all other Rewrite statements:

#--- 403 Forbidden file ---
ErrorDocument 403 /403.html

#--- enable Rewrite engine ---
RewriteEngine on
RewriteBase /

#--- block bad bots ---
# see: http://www.askapache.com/htaccess/blocking-bad-bots-and-scrapers-with-htaccess.html

RewriteRule ^403\.html$ - [L]

# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) [NC,OR]

# STARTS WITH WEB
RewriteCond %{HTTP_USER_AGENT} ^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) [NC,OR]

# ANYWHERE IN UA -- GREEDY REGEX
RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]

# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule . - [F,L]

(download this file)

The statement "ErrorDocument 403 /403.html" causes all banned users to be redirected to /403.html so you should create a simple 403.html file that just reports the user has been banned. Keep it simple without any images or external CSS references (those files would be blocked).

<html>
<head><title>403 Forbidden</title></head>
<body>
<h1>403 Forbidden</h1>
<p>Access forbidden.</p>
</body>
</html>

(download this file)

This .htaccess file is based on a post at AskApache.com. The same concept is also mentioned in a post at JavasSriptKit.com.

It is also possible to ban spiders/spammers/users based on their IP address. For example, you could ban access from IP addresses located in China. I'll cover that topic in a subsequent post.

Redirect domain.com to www.domain.com

On some websites, a user can access domain.com or www.domain.com -- notice the www. prefix.

The problem is that some search engines may see http://domain.com and http://www.domain.com as separate and consider the content to be duplicate content.

Also, some webmasters prefer users to access the host qualified domain (www.domain.com) rather than just the simple domain (domain.com)

By adding two lines to your website's .htaccess file, you can automatically redirect the user from domain.com to www.domain.com. This redirection works for all files, not just the homepage.

In your root .htaccess file, above all other RewriteRule and RewriteCond statements, add lines like these:

#--- ensure www domain prefix ---
RewriteCond %{HTTP_HOST} ^domain\.com$ [NC]
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]

Replace domain with your domain. In the RewriteCond, the \ (before the . of .com) is required otherwise the dot would match any character.

So what do these two lines do?

The RewriteCond (a rewrite condition) tests the HTTP_HOST (what domain is being accessed) and compares it to domain.com. If that condition is met, then the next statement is evaluated. If that condition is not met, then the RewriteRule is skipped.

The RewriteRule takes whatever path is being accessed and redirects to that path at the full domain. For example, /something.html would be redirected to http://www.domain.com/something.html

Note: If you have no root .htaccess or it has no RewriteRule statements, then somewhere above the preceding RewriteCond/RewriteRule statements, you also need these two statements to ensure that the Rewrite engine is enabled:

#--- enable Rewrite engine ---
RewriteEngine on
RewriteBase /

So, if you are using one of our DySE scripts, such as DySE::StubHub, then your .htaccess file might look something like this:

#--- enable Rewrite engine ---
RewriteEngine on
RewriteBase /

#--- ensure www domain prefix ---
RewriteCond %{HTTP_HOST} ^domain\.com$ [NC]
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]

#--- DySE::StubHub at root / ---
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.+$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ cgi-bin/dyse/view.pl?merch=stubhub&dir=&path=$1 [L]

The order of these three blocks of statements is important. You want the test of the domain name to be done before all other RewriteCond/RewriteRule statements.