Apache split logs utility

Too many virtual Host calls for' n' no. of log files, which is OK for a small number of virtual hosts, but if the number of hosts is very large, it can be complicated to manage and also call for insufficient file descriptors.

I decided to implement apache split_log utility for some of my web servers cluster, just wanted to share with you.

Why:

In case of multiple verticals since we have 14 vertical(virtual hosts) ( 14*2 log file active at one time for one httpd process), sometimes I am running out of available fd’s even after increasing it to a considerable good amount.

#ls –l /proc/16958//fd/

lr-x------ 1 root root 64 Feb 4 16:28 0 -> /dev/null
l-wx------ 1 root root 64 Feb 4 16:28 1 -> /dev/null
l-wx------ 1 root root 64 Feb 4 16:28 10 -> /usr/local/apache2/logs/example1.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 11 -> /usr/local/apache2/logs/example2.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 12 -> /usr/local/apache2/logs/apps.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 13 -> /usr/local/apache2/logs/data.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 14 -> /usr/local/apache2/logs/example3.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 15 -> /usr/local/apache2/logs/adsense.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 16 -> /usr/local/apache2/logs/example4.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 17 -> /usr/local/apache2/logs/music.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 18 -> /usr/local/apache2/logs/example5.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 19 -> /usr/local/apache2/logs/example6.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 2 -> /usr/local/apache2/logs/error.log
l-wx------ 1 root root 64 Feb 4 16:28 20 -> /usr/local/apache2/logs/example7.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 21 -> /usr/local/apache2/logs/marketing.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 22 -> /usr/local/apache2/logs/example8.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 23 -> /usr/local/apache2/logs/opendocs.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 24 -> /usr/local/apache2/logs/openbooks.domain.com-access_log
l-wx------ 1 root root 64 Feb 4 16:28 25 -> /usr/local/apache2/logs/example9.domain.com-access.log

# ls –l /proc/23060/fd/ | wc -l
48

Now if I have 500 similar connections, I need minimum 500*48=24,000 only for apache. Now I have entire system/ application process aside running therein. Which needs more fd’s, which I never know how much they need, as environment is dynamic.

#Edit htpd.conf

LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" virt_log

Note: “%v” does the trick.

Add:

ErrorLog logs/virt_error_log
CustomLog logs/virt_access_log virt_log

And during night hours split the logs, with apache provided perl script.

Perl split-logfile < /logs /virt_access_log virt_log, It will split all access files based on Vhost names.

Testing:

I had two vhosts viz. (1) marketing.domain.com and (2) adsense.domain.com

And after running the command I got, two log files

marketing.domain.com.log
adsense.domain.com.log

Cheers!!

Happy Splitting........