(First, this is a test post from Semagic behind the work firewall…)<\/p>\n
Okay, so I think I’ve got a good way of processing the logs from websites that I host. I’m going to document how I do it here for myself and anyone else who might happen to care.<\/p>\n
First off, I’m using Apache 2.0<\/a>, cronolog<\/a> to handle log files (essentially rotation… but not), and Webalizer<\/a> to parse the files and provide reports.<\/p>\n First, each Apache vhost is logged via a pipe to cronolog<\/font> so that it makes subdirectories for Year and Month, naming the file with Year-Month-Day-access.log. For example, today’s file as logged below would be \/var\/data\/wwwlogs\/default\/2005\/08\/2005-08-22-access.log<\/font>. This is done by CustomLog<\/font> and ErrorLog<\/font> lines for each vhost which are similar to this:<\/p>\n CustomLog “|\/usr\/local\/sbin\/cronolog \/var\/data\/wwwlogs\/default\/%Y\/%m\/%Y-%m-%d-access.log” combined That alone will make it easier to look through logfiles, should the need arise.<\/p>\n Next, a script is set to run once per night which first runs Webazolver on all the log files in order to build a cache of resolved hostnames and then runs Webalizer itself, using that cache. File processing is done with the -p<\/font> argument in order to provide incremental parsing. As each log file should<\/em> only be parsed once, the end result should be that each day’s set of data is added to the collective reports.<\/p>\n Here is the script which I call run_webalizer.sh<\/font> and will run via cron each morning (around 12:30am local time, I’d imagine):<\/p>\n #!\/bin\/sh<\/p>\n # First, run webazolver to resolve all IPs # Run webalizer with all config files… Well, hopefully some people will find this useful. It appears that it’ll work fine for now. I guess I’ll know for sure after a few days…<\/p>\n","protected":false},"excerpt":{"rendered":" (First, this is a test post from Semagic behind the work firewall…) Okay, so I think I’ve got a good way of processing the logs from websites that I host. I’m going to document how I do it here for myself and anyone else who might happen to care. First off, I’m using Apache 2.0,\u2026<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13,34],"tags":[],"class_list":["post-7594","post","type-post","status-publish","format-standard","hentry","category-computers","category-moved-from-livejournal"],"_links":{"self":[{"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/posts\/7594","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/comments?post=7594"}],"version-history":[{"count":1,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/posts\/7594\/revisions"}],"predecessor-version":[{"id":14034,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/posts\/7594\/revisions\/14034"}],"wp:attachment":[{"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/media?parent=7594"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/categories?post=7594"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nuxx.net\/blog\/wp-json\/wp\/v2\/tags?post=7594"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nErrorLog “|\/usr\/local\/sbin\/cronolog \/var\/data\/wwwlogs\/default\/%Y\/%m\/%Y-%m-%d-errors.log”<\/font><\/p><\/blockquote>\n
\nfor i in \/var\/data\/wwwlogs\/*
\n do CURLOG=`date -v-1d +”$i\/%Y\/%m\/%Y-%m-%d-access.log”`
\n \/usr\/local\/bin\/webazolver -Q -p -N 10 -D \/var\/db\/webalizer_cache.db $CURLOG
\ndone<\/p>\n
\nfor i in \/var\/data\/wwwlogs\/*
\n do VHOST=`echo $i | cut -f5 -d\\\/`
\n CURLOG=`date -v-1d +”$i\/%Y\/%m\/%Y-%m-%d-access.log”`
\n \/usr\/local\/bin\/webalizer -Q -p -n $VHOST -o \/var\/data\/www\/admin\/webalizer\/$VHOST -D \/var\/db\/webalizer_cache.db -N 10 -r $VHOST\\\/ -s \\*$VHOST $CURLOG
\ndone<\/font><\/p><\/blockquote>\n