a horror story in the systemd house of horror

You really don't need to daemonize. Really.

The cast

Ali Erdinç Köroglu after very earnestly explaining the use of daemonize on System 5 rc systems promptly carried that knowledge over to systemd operating systems, very earnestly explaining how to use daemonize in a service unit and then "the best way" to daemonize things run from systemd, because systemd service units "are not able to automatically restart a crashed process".

Ali Erdinç Köroglu mentions that one reason to choose daemonize is that start-stop-daemon isn't available outwith Debian Linux and its derivatives. Ralf Geschke uses start-stop-daemon in a service unit, nonetheless.

The rather sad thing is that none of this was actually necessary at all.

The horror story

These are all examples of the error of carrying over knowledge of existing systems onto systemd. "I must use start-stop-daemon.", thinks the poor system administrator. "One always uses start-stop-daemon It's used in the examples in the Debian Policy Manual, for pity's sake!" Similarly, the administrator thinks "systemd units are just like System 5 rc scripts, so they must suffer from the same problems of not auto-restarting services and not properly tracking stuff. So I must use extra tools to do that."

M. Geschke's service unit actually introduces bugs and misbehaviours that systemd (and other service managers that track processes properly, by dint of simply remembering what they forked) does away with.

As a consequence, the service cannot be a plain old Type=simple. Instead, it is, completely unnecessarily:

Type=forking
PIDFile=/var/run/redis/redis.pid

M. Köroglu's use of daemonize and supervisord is similarly entirely unnecessary, and in its entirety duplicates things that systemd can and does do. And again, it ends up actually introducing misbehaviours that systemd gets rid of.

M. Köroglu uses daemonize to redirect the standard output and standard error of a Java program. But this is a known-bad way to go about logging. One cannot size-cap the log file; one cannot rotate the log file; one cannot shrink the log file. It grows, as one file, forever until the dæmon is shut down. This is ridiculous when we have systemd. systemd can send standard output and standard error to multiple log files that — whatever else one may think of them — are most definitely size capped and rotatable.

the first rule for migrating to systemd

At this point, in 2015, it's most likely that someone has already done it.

systemd has been around for some years. And there has been a whole cottage industry of people writing unit files and publishing them. GitHub, in particular, seems to attract repositories of collections of service units.

systemd can automatically restart crashed processes. systemd can run Java programs directly. systemd can run dæmons under the aegeses of unprivileged user accounts. systemd can change directory. The knowledge that one should carry over is from experience with things like daemontools family toolsets. In the daemontools family world and indeed the world of fully fledged service managers in general, people have known for decades that the service manager both can and does do these things; that the dæmon does not; and that most especially one doesn't bring over old System 5 rc bodges when one has dæmon programs that already do things properly and do not attempt to mistakenly "daemonize" themselves.

You really don't need to daemonize. Really.

It is especially easy to do things right when the first rule of migrating to systemd applies. If M. Geshke had only looked elsewhere on GitHub he would have seen that people have long since written real redis.service units, like this one by Remi Collet written the year before (where the directory creation is done through a separate tmpfiles control file). The Arch Linux people have a redis.service that demonstrates running under the aegises of unprivileged user accounts and more besides (including the RuntimeDirectory setting), along with a reminder that you really don't need to daemonize, so you turn the daemonize option in redis off, too.

The fixtures service only needs the one file, yet another example of the how-to-start-a-Java-program-from-systemd pattern, which simply tells systemd to restart the dæmon on failure (a 1 setting addition to what the unit file already had) without requiring an entire supervisord installation just for that:

[Unit]
Description=Fixtures Service
After=syslog.target
After=network.target
 
[Service]
User=pronet
WorkingDirectory=/opt/pronet/fixtures
ExecStart=/usr/java/jdk1.7.0_71/bin/java \
-Dfile.encoding=UTF-8 \
-Dproject.properties=/opt/pronet/fixtures/fixtures.properties \
-Dlog4j.configuration=file:/opt/pronet/fixtures/fixtures-log.properties \
-jar /opt/pronet/fixtures/fixtures.jar
Restart=always
 
[Install]
WantedBy=multi-user.target

The systemd people prefer Restart=on-failure on the grounds that if a system administrator manually kills the process xe is obviously expecting the service to die, because that's what killing a dæmon process used to do on systems without service managers, so it should. This is one of many examples where this thinking is shown up as too conservative. Ali Erdinç Köroglu, like many other people, wants the service to restart, and, ironically, doesn't think that systemd is up to the job.


© Copyright 2015 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.