Recently I set up a server to which all other ResTek machines may log. OpenBSD was chosen as the operating system because it is simple, secure, and stable.

Not all of the required functionality is included in the stock syslogd (I’m glad), so I also installed the syslog-ng package (available as an OpenBSD package). Syslog-ng is nice because it allows you to automatically organize all of the incoming logs.

Syslog-ng is very flexible, but configuration was short and sweet in this case. These are the most important lines of the configuration file:

destination files {
        file("/var/log/hosts/$HOST/$YEAR/$MONTH/$DAY/$HOST-$FACILITY-$LEVEL-$YEAR-$MONTH-$DAY.log"
                owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)
        );
};

destination tenshi {
        udp("127.0.0.1", port(515));
};

log { source(all); destination(files); destination(tenshi); };

All of the directories/files are created as needed, which is awesome. In addition to logging all of the incoming data to files (organized as seen above), logs are also sent to port 515 on localhost, a.k.a. “tenshi.” What is “tenshi” you might ask?

Tenshi is a log monitoring program, designed to watch one or more log files for lines matching user defined regular expressions and report on the matches. The regular expressions are assigned to queues which have an alert interval and a list of mail recipients.

In our case, I didn’t want to have Tenshi monitor too many files and have to deal with opening/closing them as dates changed, so I set up Tenshi to simply listen on port 515 on localhost. All of the logs are sent to this location, and Tenshi processes them as if it were watching a bunch of log files. One place to watch messages from a dozen machines!

Here are some of the important lines from Tenshi’s configuration file, tenshi.conf:

set listen 127.0.0.1:515
...
snort ^snort:

critical ^.*failed.*

critical ^(?:httpd|php): PHP Warning:
critical ^(?:httpd|php): PHP Fatal error:
critical ^(?:httpd|php): PHP Parse error:

Each of these messages is grouped together and mailed out at various intervals (in the case of “critical” messages, messages are e-mailed immediately). Recipients of the messages (anyone on the mailing list…) organizes them into different folders as well.

How do messages get to the central logserver? Each of the machines (including the logserver itself!) has an extra line in syslogd.conf:

*.*                                                     @loghost

All messages logged to syslogd are, as a result, forwarded to the loghost in addition to being logged locally.

Pretty neat huh?