In one of my previous post, I discussed and showed how to stop Bad Bots from crawling your website using .htaccess. This particular method is highly useful if you are running an Apache web server. The question is, what about if your server is not using Apache? Well, there are effective methods are there. We can use HTML meta tag as well. Today on this post, this what exactly I will be discussing. Meta tag generally resides with in the <head> and </head> tag on your HTML page. So, in order stop any specific bot (i.e. Gigabot) from crawling our site we can use the following snippet.
In this example, we are asking "Gogabot" not to crawl our pages. But the question is, is it really possible to mention every single bot within meta tag like this? well, it is possible but that's not the best choice. Especially if you other options available. Let's take a look at the following meta tag.
<meta name="Gigabot" content="noindex">
Using this tag will stop every single bot from crawling your sites. This tag could particularly be every good if you do not want any bot to crawl your page whatsoever. Now, using this meta tag comes with its down side. Note: Not allowing any bot will stop good bots (i.e. Google) from crawling your pages as well. Most probably this is not what you want either unless you really want your site to be private. Perhaps the best option would be to check your server or visitor log and identify specific harmful bots and stop them manually, using the meta tag I just mentioned above. References: Meta Tag, HTML Meta Tag, robotstxt.org, BotReports.
<meta name="robots" content="noindex">