Today we had our first crash since we moved to the new server 7 months ago. Actually, the server wasn’t exactly crashed but it had come to a point where it was unresponsive enough to be unable to serve HTTP pages (and to make it a pain to connect to SSH). I wasn’t really in the mood so I didn’t take a screenshot, but top reported an impressive 90+ load average (NB: this server currently only has one core). Anyway, basically it seemed that the server was flooded with apache2 processes. Webmin reported a full use of our (massive ;)) 1 GiB of RAM as well as almost full use of our (twice as massive) virtual memory. I thought maybe we were being DOSed, although I don’t really see who would want to do that at the moment. I first stopped Apache, which basically solved the problem. But before starting it again I had a look at the logs… and in /var/log/apache2/access.log I found a massive amount of those:
127.0.0.1 – – [27/Jul/2011:09:55:22 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:55:23 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:55:24 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:55:52 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:55:54 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:55:55 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:55:56 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:55:58 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:56:05 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
127.0.0.1 – – [27/Jul/2011:09:56:06 +0200] “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)”
I remember we already had issues with these internal dummy things on the previous server, but because software there was getting old our response to the problem was just, well, to move to the new server with all updated software and stuff. This time I decided to try to deal with the problem for real.
After some quick searches, I found out that internal dummy connections are used to wake up child processes. I’m not sure how all this works, but the key point here is that in the end, sometimes those calls may hit some of your “real” pages, and may also corrupt their cached version. Long story short, the request may end up more CPU-demanding than it should. That problem seems to have been solved in Apache 2.2.6, but although we are running a much later version I thought it wouldn’t hurt to still apply the work-around. Basically, all you need to do is to make sure that the dummy connections get a 403 error. So, just 2 lines in your configuration files (NB: it requires mod_rewrite to be enabled) (source):
RewriteCond %{HTTP_USER_AGENT} ^.*internal\ dummy\ connection.*$ [NC]
RewriteRule .* – [F,L]
And voilà, internal dummy connections should now be processed effortlessly. Don’t forget to apply configuration changes.
References (if you need more details you’ll find quite a few there):
Cool, I wonder if shared hosting providers allow these types of configuration changes for Apache?
Usually in shared hosting you can’t change config files, but for those particular settings putting them in an .htaccess file should work fine. And anyway, this thing is only a problem for the resources used by the server, so probably your hosting provider will do their best to tweak it themselves!
I’ve done the fixes and I’m still crashing…. I’ve even managed to update apache to 2.2.22 but the issue still persists. My dedicated server i5 with 4 gigs of ram is crashing every night and I can’t seem to stop it. I’ve created this thread, http://forums.digitalpoint.com/showthread.php?t=2465142 to try and get some help, sadly I’m a marketer not a server admin but I’m trying to learn as I go. Seems updating apache using CentOS has it’s issues we had a hard time even getting it to 2.2.22
Sorry I really don’t know better than what I wrote in this post. I’m currently using Apache 2.2.16 on Debian 6.0.4, and those dummy connections don’t appear in the logs anymore, without any special tweaking on my end (that’s a quite new install, so I remember doing nothing special about those dummy things)… That’s really some weird issue… :/
For me on Debian squeeze with apache2 .2.16 using the mod rewrite is not solving the annoying message 😐
Man these are really annoying when they come up in the logs.