newsyslog

The Moving Finger writes; and, having writ,
Moves on: nor all thy Piety nor Wit
Shall lure it back to cancel half a Line,
Nor all thy Tears wash out a Word of it.

Omar Khayyam, The Rubayyat

Description

This was one of the first large C programs that I wrote entirely from scratch, and one of my first projects when I came to work for Stanford. It's been reworked a few times since then, though, so it mostly doesn't show.

Before I wrote this program, we were using hacked versions of Sun's standard log rotation shell scripts to save system logs. The purpose of this program was to provide portable log rotation for all of our servers and cluster machines, for the most part writing the log files into AFS. I wrote it to be completely stand-alone and not depend on any external programs, since we wanted it to work even if AFS was down (and nearly all software for our machines is in AFS).

All log files rotated by newsyslog are compressed, with the choice of either gzip-style zlib compression or bzip2 compression. newsyslog also has the ability to run an analyze program on the log file (usually useful to scan the log for anomalous entries and report them, perhaps by sending a report in e-mail), and to run a filter program on the log before archiving it and only archiving the output from the filter (so that the uninteresting lines can be stripped out before archiving). It also has the standard features that you'd expect in a log rotation program, such as being able to save a given number of old copies of a log and running commands to stop and restart the service around the log rotation.

Since I wrote this program, various other programs that do similar things have shown up, most notably logrotate (which is widely used in Linux distributions). The main reason to use newsyslog instead of one of those programs is that it's simpler, more self-contained, recovers smoothly from failed log rotations when the destination directory is temporarily unavailable, and has support for doing a minimum of file operations to save an archived log (so that we can archive logs into AFS directories with very restrictive ACLs). It is not as flexible or as featureful as other log rotation programs, however.

For the program I use to analyze syslog logs after rotating them, see filter-syslog.

Requirements

newsyslog is written in C, so you'll need a C compiler. You'll also need a compression library, at least one of zlib or the library that comes with bzip2. For support of the -b option, bzip2 is required, and if zlib is not installed, the -b option must always be used.

Note that zlib and bzip2 are included with most, if not all, Linux distributions, so if you're on Linux, check your distribution before taking the time to download and build them yourself. To build this package, you'll have to have the headers and libraries installed, which means you may have to install the -dev or -devel version of the package as well as the basic version.

If you plan to hack on the newsyslog sources, you'll also need gcc (since the dependency generation relies on it), bison, flex, pod2man, and Autoconf 2.57 or later, depending on what you change. None of those additional programs are required to build the distribution (the output of bison and flex is included in the distribution).

Download

The distribution:

An archive of older releases is also available.

Documentation:

• README
• newsyslog manual page
• Installation instructions
• To-do list
• Change summary and full log

A Debian package is available from my personal repository.

newsyslog is maintained using the Git version control system. To check out the current development tree, clone:

    git://git.eyrie.org/system/newsyslog.git

You can also browse the current development source.

License

Copyright 1997, 1998, 1999, 2000, 2002, 2004, 2005 Board of Trustees, Leland Stanford Jr. University.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.