Common traits in network monitoring and DoS

| | No TrackBacks
The Siege of Antioch

Pros and cons of monitoring

The disadvantage of not monitoring network resources is obvious: in case of any connectivity or functionality problem, you are not warned, thus a number of unpleasant consequences may follow.

All right, so we accept we need monitoring. The next question is, whether this can harm your resources (site, for example) in any way? The not so obvious answer is yes, it can.

Let's imagine we have a site and let's analyze who accesses it and how the site can react to this.

Bots, your zealous readers

Who's reading your site, mostly? Unless yours is a popular blog with thousands of readers, the answer is: bots. Search spiders, RSS bots, whatever else. Human readers can at times be in severe minority.

Spiders can be a nuisance, especially new ones, mostly misbehaving. They do not always obey robots.txt rules, thus can exhaust your bandwidth or overload server, and cause many other malfunctions. However, these often supply their identifier ('User-Agent' header) and can be told from all the other readers, especially if IP addresses of crawlers are known.

Faceless monitoring

Here we come to a very interesting conclusion: if a monitoring process doesn't behave like a browser, or at least like a well-known bot, if it sends no HTTP headers and connects to a server periodically and at known time intervals, it can be viewed as harmful.

Anonymous spiders, those harvesting email addresses and other information, are seldom written in a well-behaving manner. They do not respect robots.txt (of course),they have a habit of overwhelming server with requests.

In other words, if your monitoring process doesn't introduce itself with User-Agent, does access the site too often and doesn't throttle data transfer speed, it can be viewed as data harvester or even an attacker. A common DoS is exhausting bandwidth, when a number of processes request large pages many a time in quick sequence.

So, if you plan to use monitoring HTTP/HTTPS resources, make sure that
— your monitoring software supplies HTTP headers, providing unique id
— your monitoring software does throttle the connection speed and does not exhaust too much of channel speed
— your monitoring software adds small random intervals toits polling schedule

No TrackBacks

TrackBack URL: /blog/mt-tb.cgi/105

blog comments powered by Disqus

About this Entry

This page contains a single entry by Konstantin Boyandin published on June 4, 2010 5:16 PM.

Monitoring tools: learn more from reinventing the wheel was the previous entry in this blog.

Data cobweb: let computers talk to computers is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.