About
nuxx.net runs lighttpd, a high performance, low overhead web server (httpd) to serve up content for a number of domains. With the number of virtual hosts (vhosts) on the server being owned by a number of different users, the configuration used for lighttpd is a bit different from the standard setup of one server, one website.
While an overall understanding of how lighttpd is configured is beyond the scope of this document, here I aim to address how I have configured lighttpd along with examples. These cover running PHP as separate users for privilege separation, automatically rotating logs which users cannot delete, and more. For more information please consult the official lighttpd Docs.
If you have any questions, comments, or corrections, please do not hesitate to contact me.
User Structure / vhosts
lighttpd's main config file, lighttpd.conf contains an include statement which points to a config file for each vhost. Each of the included config files are in a lighttpd-includes directory beneath the directory where the main lighttpd.conf is stored. This setup allows each vhost config to be turned on and off via the main config, maintained separately from other vhosts, and generated from a template. For example, from the lighttpd.conf:
include "lighttpd-includes/siteone.com.conf" include "lighttpd-includes/sitetwo.net.conf" include "lighttpd-includes/sitethree.org.conf" include "lighttpd-includes/sitefour.us.conf" #include "lighttpd-includes/sitedisabled.com.conf"
Unfortunately, as of lighttpd v1.4.19, wildcard includes such as what follows here do not work. If they did, it would make enabling / disabling vhosts as easy as renaming a file and restarting lighttpd:
include "lighttpd-includes/*.conf" # Does not work.
Each vhost's config file then points to one or more website root directories located under a particular user's home directory. Each of these directories are set to 750 and user:usergroup For example, here is a directory listing of ~enduser/www:
enduser@server:~/www> ls -als total 10 2 drwxr-x--- 5 enduser enduser 512 Sep 28 2006 . 2 drwxr-xr-x 15 enduser enduser 1024 Aug 7 20:38 .. 2 drwxr-x--- 2 enduser enduser 512 Sep 18 2006 default 2 drwxr-x--- 3 enduser enduser 512 Jul 31 17:02 siteone.com 2 drwxr-x--- 15 enduser enduser 512 Aug 12 20:00 sitetwo.net enduser@server:~/www>
Sample lighttpd.conf and vhost config files may be found in the Example Files section.
As part of this setup the user www must be a member of each user's group. For example, if there is a user named enduser whose login group is enduser and this user has a vhost, www must be a member of this user's login group. As seen in /etc/group:
enduser:*:1019:www
Default Site
As a catch-all for vhosts not defined elsewhere in the config, it is suggested that one set up a 'default' site. This can contain anything from contact info to a simple go-away page. For example, the default site on nuxx.net contains a single index.html which is as follows:
<title>Nope. Sorry.</title> <html><font face=verdana>Sorry, nothing to see here. Please move along.</font></html>
This default site is configured outside of any vhosts lighttpd.conf:
server.document-root = "/home/enduser/www/default/" accesslog.filename = "|/usr/local/sbin/cronolog -u www -g www /home/enduser/logs/www/default_%Y-%m-%d_access.log"
Per-User PHP
In order to have appropriate privilege separation PHP runs as a FastCGI under the username which owns each vhost. For example, in the above example with siteone.com and sitetwo.net, multiple copies of php-cgi are running under username enduser. These are accessed through a socket in /var/run/php-fastcgi/enduser:
enduser@banstyle:/var/run/php-fastcgi/enduser> ls -als total 6 2 drwxr-xr-x 2 enduser www 512 Aug 21 17:22 . 2 drwxr-xr-x 12 root wheel 512 Aug 21 16:40 .. 2 -rw-r--r-- 1 root www 4 Aug 21 17:22 enduser-php-fastcgi.pid 0 srwxrwx--- 1 enduser www 0 Aug 21 17:22 enduser-php-fastcgi.sock enduser@banstyle:/var/run/php-fastcgi/enduser>
To run PHP apps the vhost config file then just points to the socket:
fastcgi.server = ( ".php" =>
(
( "socket" => "/var/run/php-fastcgi/enduser/enduser-php-fastcgi.sock",
"check-local" => "disable",
"broken-scriptfilename" => "enable"
)
)
)
This socket and the PID file are created by per-user user-php-fastcgi.sh scripts located in /usr/local/sbin/php-fastcgi. These scripts are created from this template and automatically spawned by fastcgi.sh which is placed in /usr/local/etc/rc.d so that it automatically runs on boot.
The main advantage of this configuration is that PHP processes run as the user whose vhost hosts the PHP app. This mitigates damage which can be caused by an exploited application by generally limiting the rights gained by the attacker to those of a common user on the machine. Of course, an attack on a PHP application could be combined with a local exploit to gain root-level permissions, but this configuration does add a huge barrier.
Logging
Log files are written using cronolog into ~/logs/www/. Using cronolog allows the log files to be easily rotated out each day. By using a line similar to this in vhost config the logs are automatically written with a separate file each day:
accesslog.filename = "|/usr/local/sbin/cronolog -u www -g www /home/enduser/logs/www/siteone.com_%Y-%m-%d_access.log"
The directory for the logs, ~/logs/www is set chmod 750 and chown www:enduser, which results in the log files inside being writable by www, which is what lighttpd and cronolog run as and readable by enduser. This prevents users or exploited applications from deleting lighttpd's logs, which is quite useful when trying to understand how an account compromise occurred.
Note that the server.errorlog directive can only appear once in your lighttpd config, therefore it is recommended that it be placed in the main lighttpd.conf and point to the default site. If it appears multiple times the last entry for it will be the one used.
Unfortunately, as of v1.4.19 the server.errorlog directive does not support a pipe and thus doesn't work with cronolog.
Webalizer
While log files alone can be useful, it's often nicer to have them parsed and interpreted. In order to provide basic information Webalizer runs once each night to handle the previous day's log files. This is done automatically via webalizer_daily.sh called run by cron each night at 10 minutes after midnight:
10 0 * * * /usr/local/sbin/webalizer_daily.sh
Additionally, webalizer_daily.sh also runs webazolver to first resolve and cache IPs logged, creates output directories as needed, and deletes log files which more than 30 days old.
App-specific Configurations
Many open source web-based applications are written to write their own rewrite rules for Apache's .htaccess, but as lighttpd doesn't use these files, rewrite rules must be manually configured in the vhost config. Here are rewrite rules used on nuxx.net to facilitate some of the applications running on the site:
Gallery 2
url.rewrite = (
"^/gallery/v/(\?.+|\ .)?$" => "/gallery/main.php?g2_view=core.ShowItem",
"^/gallery/admin[/?]*(.*)$" => "/gallery/main.php?g2_view=core.SiteAdmin&$1",
"^/gallery/d/([0-9]+)-([0-9]+)/([^\/]+)(\?|\ )?(.*)$" =>
"/gallery/main.php?g2_view=core.DownloadItem&g2_itemId=$1&g2_serialNumber=$2&$3",
"^/gallery/v/([^?]+)/slideshow.html" =>
"/gallery/main.php?g2_view=slideshow.Slideshow&g2_path=$1",
"^/gallery/v/([^?]+)(\?|\ )?(.*)$" =>
"/gallery/main.php?g2_view=core.ShowItem&g2_path=$1&$3",
"^/gallery/c/add/([0-9]+).html" =>
"/gallery/main.php?g2_view=comment.AddComment&g2_itemId=$1",
"^/gallery/c/view/([0-9]+).html" =>
"/gallery/main.php?g2_view=comment.ShowAllComments&g2_itemId=$1",
"^/gallery/p/(.+)" =>
"/gallery/main.php?g2_controller=permalinks.Redirect&g2_filename=$1"
)
MediaWiki
url.rewrite = (
# Make MediaWiki the main site
"^/$" => "/w/index.php",
# Remove index.php and make URLs appear to be at /wiki
"^/wiki$" => "/w/index.php",
"^/wiki/([^?]*)(?:\?(.*))?" => "/w/index.php?title=$1&$2",
# Google Sitemaps to the right place.
"^/sitemap.xml" => "/w/sitemap.xml.php"
)
Wordpress
url.rewrite = (
"^/?$" => "/blog/index.php",
"^/blog/(wp-.+)$" => "$0",
"^/blog/xmlrpc.php" => "$0",
"^/blog/sitemap.xml" => "$0",
"^/blog/(.+)/?$" => "/blog/index.php/$1"
)
Limitations
The biggest limitation I've found with using lighttpd is its lack of dot.files, such as the widely used .htaccess. This means that changes access rules, rewriting rules, and more have to be made in the vhost config with a restart of the server taking place after the changes are made. This also prevents applications such as Gallery from writing their own rewrite rules.
However, in practice I've found that for stable sites it is not difficult to add some application specific rules to the vhost configs. In my experience sites the sites I host do not change such rules frequently enough for this to be a problem.
Fun with lighttpd
If you have a problem with people inline linking (also known as hotlinking) files hosted on your site in a less than desirable manner, there are a number of ways to combat this. Here is a particularly fun example, which replaces all JPEGs, PNGs, and GIFs linked by MySpace users with another more interesting image.
$HTTP["referer"] =~ "^.*profile.myspace.com.*$" {
url.rewrite = ("(?i)(/.*\.(jpe?g))$" => "/images/oops.jpg",
"(?i)(/.*\.(png))$" => "/images/oops.png",
"(?i)(/.*\.(gif))$" => "/images/oops.gif" )
}
Example Files
user-php-fastcgi.sh
#!/bin/sh ## Some locations... SPAWNFCGI="/usr/local/bin/spawn-fcgi" FCGIPROGRAM="/usr/local/bin/php-cgi" FCGISOCKET="/var/run/php-fastcgi/username/username-php-fastcgi.sock" FCGIPID="/var/run/php-fastcgi/username/username-php-fastcgi.pid" ## Number of PHP children to spawn. Default is 5, minimum is 2, max is 256. PHP_FCGI_CHILDREN=5 ## Number of request server by a single php-process until is will be restarted PHP_FCGI_MAX_REQUESTS=1000 ## IP adresses where PHP should access server connections from FCGI_WEB_SERVER_ADDRS="127.0.0.1" # Environment variables which are allowed to be passed to FastCGI ALLOWED_ENV="PATH USER" ## Username and Groupname to run as. Generally will be the same. USERID=username GROUPID=usergroup ## Want a custom php.ini? Define it here. PHPRC="" #### ## Nothing more to do below here... #### export PHP_FCGI_MAX_REQUESTS export FCGI_WEB_SERVER_ADDRS export PHPRC env -i PHP_FCGI_MAX_REQUESTS=$PHP_FCGI_MAX_REQUESTS \ FCGI_WEB_SERVER_ADDRS=FCGI_WEB_SERVER_ADDRS PHPRC=PHPRC \ $SPAWNFCGI -f $FCGIPROGRAM -s $FCGISOCKET -P $FCGIPID \ -u $USERID -g $GROUPID -C $PHP_FCGI_CHILDREN chmod 770 $FCGISOCKET
lighttpd.conf
(Note that this file is as used in production on nuxx.net, but with most of the defaults, or non-useful commented out items removed. Please reference the latest development lighttpd.conf and official configuration documentation for more information.)
# lighttpd configuration file
## modules to load
# at least mod_access and mod_accesslog should be loaded
# all other module should only be loaded if really neccesary
# - saves some time
# - saves memory
server.modules = (
"mod_rewrite",
"mod_redirect",
# "mod_alias",
"mod_access",
# "mod_cml",
# "mod_trigger_b4_dl",
"mod_auth",
"mod_status",
# "mod_setenv",
"mod_fastcgi",
# "mod_proxy",
# "mod_simple_vhost",
# "mod_evhost",
# "mod_userdir",
"mod_cgi",
# "mod_compress",
"mod_ssi",
"mod_usertrack",
# "mod_expire",
# "mod_secdownload",
# "mod_rrdtool",
"mod_accesslog",
"mod_flv_streaming" )
## a static document-root, for virtual-hosting take look at the
## server.virtual-* options
server.document-root = "/home/user/www/default/"
## where to send error-messages to
server.errorlog = "/home/user/logs/www/default_error.log"
# files to check for if .../ is requested
index-file.names = ( "index.html", "index.php",
"index.htm", "default.htm" )
## set the event-handler (read the performance section in the manual)
server.event-handler = "freebsd-kqueue" # needed on OS X
# mimetype mapping
mimetype.assign = (
".pdf" => "application/pdf",
".sig" => "application/pgp-signature",
".spl" => "application/futuresplash",
".class" => "application/octet-stream",
".ps" => "application/postscript",
".torrent" => "application/x-bittorrent",
".dvi" => "application/x-dvi",
".gz" => "application/x-gzip",
".pac" => "application/x-ns-proxy-autoconfig",
".swf" => "application/x-shockwave-flash",
".tar.gz" => "application/x-tgz",
".tgz" => "application/x-tgz",
".tar" => "application/x-tar",
".zip" => "application/zip",
".mp3" => "audio/mpeg",
".m4a" => "audio/mpeg",
".m3u" => "audio/x-mpegurl",
".wma" => "audio/x-ms-wma",
".wax" => "audio/x-ms-wax",
".ogg" => "application/ogg",
".wav" => "audio/x-wav",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".png" => "image/png",
".xbm" => "image/x-xbitmap",
".xpm" => "image/x-xpixmap",
".xwd" => "image/x-xwindowdump",
".css" => "text/css",
".html" => "text/html",
".htm" => "text/html",
".js" => "text/javascript",
".asc" => "text/plain",
".c" => "text/plain",
".cpp" => "text/plain",
".log" => "text/plain",
".conf" => "text/plain",
".reg" => "text/plain",
".text" => "text/plain",
".txt" => "text/plain",
".patch" => "text/plain",
".dtd" => "text/xml",
".xml" => "text/xml",
".mp4" => "video/mp4",
".mpeg" => "video/mpeg",
".mpg" => "video/mpeg",
".mov" => "video/quicktime",
".qt" => "video/quicktime",
".avi" => "video/x-msvideo",
".asf" => "video/x-ms-asf",
".asx" => "video/x-ms-asf",
".wmv" => "video/x-ms-wmv",
".bz2" => "application/x-bzip",
".tbz" => "application/x-bzip-compressed-tar",
".tar.bz2" => "application/x-bzip-compressed-tar"
)
#### accesslog module
accesslog.filename = "|/usr/local/sbin/cronolog -u www -g www /home/user/logs/www/default_%Y-%m-%d_access.log"
## deny access the file-extensions
#
# ~ is for backupfiles from vi, emacs, joe, ...
# .inc is often used for code includes which should in general not be part
# of the document-root
url.access-deny = ( "~", ".inc" )
$HTTP["url"] =~ "\.pdf$" {
server.range-requests = "disable"
}
##
# which extensions should not be handle via static-file transfer
#
# .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
## to help the rc.scripts
server.pid-file = "/var/run/lighttpd.pid"
# chroot() to directory (default: no chroot() )
#server.chroot = "/"
## change uid to <uid> (default: don't care)
server.username = "www"
## change uid to <uid> (default: don't care)
server.groupname = "www"
#### status module
#status.status-url = "/server-status"
#status.statistics-url = "/server-statistics"
#status.config-url = "/server-config"
$HTTP["remoteip"] =~ "(127.0.0.1|204.11.33.41)" {
status.status-url = "/server-status",
status.statistics-url = "/server-statistics"
}
#### include
include "lighttpd-includes/siteone.com.conf"
include "lighttpd-includes/sitetwo.net.conf"
include "lighttpd-includes/sitethree.org.conf"
include "lighttpd-includes/sitefour.us.conf"
#include "lighttpd-includes/sitedisabled.com.conf"
php-fastcgi-user.sh
#!/bin/sh # # Just simply run all the scripts to start php-fastcgi for each user. # for i in /usr/local/sbin/php-fastcgi/*.sh do $i done
siteone.com.conf
### siteone.com
$SERVER["socket"] == ":80" {
$HTTP["host"] =~ "(^www.|^)siteone.com$" {
server.document-root = "/home/enduser/www/siteone.com"
accesslog.filename = "|/usr/local/sbin/cronolog -u www -g www /home/enduser/logs/www/siteone.com_%Y-%m-%d_access.log"
fastcgi.server = ( ".php" =>
(
( "socket" => "/var/run/php-fastcgi/enduser/enduser-php-fastcgi.sock",
"check-local" => "disable",
"broken-scriptfilename" => "enable"
)
)
)
dir-listing.activate = "enable"
}
}
webalizer_daily.sh
#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
for i in /home/*/logs/www/*_`date -v-1d +"%Y-%m-%d_access.log"`; do
# Run webazolver on each log file to compile DNS info.
/usr/local/bin/webazolver -Q -p -N 10 -D /var/db/webalizer_cache.db $i
# Build the vhost name.
VHOST=`echo $i | cut -f6 -d \/ | cut -f1 -d _`
# Create missing log dirs.
if [ ! -d /home/user2/www/admin.siteone.com/webalizer/$VHOST ]; then
mkdir /home/user2/www/admin.siteone.com/webalizer/$VHOST
fi
# Actually run webalizer.
/usr/local/bin/webalizer -Q -p -n $VHOST \
-o /home/user2/www/admin.siteone.com/webalizer/$VHOST \
-D /var/db/webalizer_cache.db -N 10 -r $VHOST -s \*$VHOST $i
done
# Delete month-old log files.
for i in /home/*/logs/www/*_`date -v-30d +"%Y-%m-%d_access.log"`; do
if [ -e $i ]
then rm $i
fi
done