a horror story in the systemd house of horror

Useless use of System 5 rc scripting with WSO2 Carbon Server

The cast

Akila Senarath discusses how to run wso2 under systemd. He provides a script that takes start and stop verbs, and a systemd service unit file that invokes that script with those verbs.

ExecStart=/<wso2serverHOME>/bin/myService.sh start
ExecStop=/<wso2serverHOME>/bin/myService.sh stop

"But, hey!", you might ask. "That systemd service unit file is only 14 lines long. How could that be horrendous?"

The horror story

Let us look at what his script does:

So it is nothing but a Poor Man's Dæmon Supervisor written (badly, as they always are) in shell script. But we already have a better dæmon supervisor. It's systemd, the thing that he is trying to run wso2 under in the first place. So there's no need for any of this. There's certainly no need for the PID file nonsense; that's exactly the sort of thing that is badly written in shell script, and it is a dangerous, rickety, and unreliable mechanism. Proper service managers have no need of it. They just remember the PID of the child that they themselves forked.

Alright, so we could at least write the service unit to invoke wso2server.sh directly. It gets worse, though.

wso2server.sh is a script that is supplied as part of WSO2 Carbon Server. Like far too many such scripts, it's difficult to see the forest for the trees. There's a lot of stuff there that cannot possibly apply on a systemd operating system, because systemd doesn't run on OS/400 or Cygwin or mingw. There's also a lot of stuff there that cannot possibly apply since we're only ever coming in here, remember, with either the start or the stop action. Strip those away, and its operation gives one an acute sense of déja vu:

So this is nothing but a Poor Man's Dæmon Supervisor written (badly, as they always are) in shell script … too. We thus have two Poor Man's Dæmon Supervisors: both maintaining their own entirely pointless, and different, PID files; and both forking children and exiting.

Akila Senarath has clearly had to fight with this, as he has bodged his service unit to erroneously state the service to be a Type=oneshot service (which it is not because it keeps running after startup) and then had to bodge on top of the bodge specifying that it is a RemainAfterExit=true service. The WSO2 people have clearly had to fight with this, too. The WSO2 doco says, and has said for several product releases now, that

"We do not recommend starting WSO2 products as a daemon, because there is a known issue that causes automatic restarts in the wrapper mode."

Well, yes, there is. It's the madness that is that Poor Man's Dæmon Supervisor, badly written in shell script. And Akila Senarath has unfortunately piled more madness on top of it.

Let us consider what happens as a result of that seemingly innocuous ExecStart line:

  1. systemd spawns myService.sh start, remembering its PID.

  2. myService.sh forks su (an abuse of su, incidentally) which because of PAM in its turn forks wso2server.sh start.

  3. In parallel:

There are three models that a systemd-managed dæmon can follow, and this doesn't match any of them.

In this setup, the actual dæmon process is the great-great-grandchild of the process that systemd knows about.

We need make a file, let us call it /etc/default/wso2, with all of the site-local environment variable settings explaining such things as where the Java Runtime Environment is this week.

JAVA_HOME=/usr/share/java/jre-x.y.z
CARBON_HOME=/usr/share/wso2
CARBON_XBOOTCLASSPATH=
CARBON_CLASSPATH=
JAVA_ENDORSED_DIRS=/usr/share/wso2/lib/endorsed:/usr/share/java/jre-x.y.z/jre/lib/endorsed:/usr/share/java/jre-x.y.z/lib/endorsed

systemd can set environment variables and run a Java program. systemd will track its PID properly, no rickety PID file nonsense required. systemd will ensure that only one dæmon is ever started at any time. systemd can be told about special exit statuses that mean a automatic restart. systemd can be told to run services as particular user accounts.

[Unit]
Description=WSO2 Carbon Server service
Wants=mysql.service
After=mysql.service

[Service]
User=wso2
WorkingDirectory=/usr/share/wso2
EnvironmentFile=-/etc/default/wso2
RestartForceExitStatus=121
ExecStart=/usr/bin/env ${JAVA_HOME}/bin/java \
-Xbootclasspath/a:${CARBON_XBOOTCLASSPATH} \
-Xms256m -Xmx1024m -XX:MaxPermSize=256m \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=${CARBON_HOME}/repository/logs/heap-dump.hprof \
$JAVA_OPTS \
-Dcom.sun.management.jmxremote \
-classpath ${CARBON_CLASSPATH} \
-Djava.endorsed.dirs=${JAVA_ENDORSED_DIRS} \
-Djava.io.tmpdir=${CARBON_HOME}/tmp \
-Dcarbon.registry.root=/ \
-Djava.command=${JAVA_HOME}/bin/java  \
-Dcarbon.home=${CARBON_HOME} \
-Dcarbon.repository=${CARBON_HOME}/repository \
-Djava.util.logging.config.file=${CARBON_HOME}/repository/conf/etc/logging-bridge.properties \
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog \
-Djava.security.egd=file:/dev/./urandom \
-Dfile.encoding=UTF8 \
org.wso2.carbon.launcher.Main

[Install]
WantedBy=multi-user.target

Of course, this invokes Java by running /usr/bin/java rather than executing the target program file directly. systemd insists that the name of the dæmon, in the systemd journal, is thus "java". Running it indirectly via /usr/bin/env, in order to cope with the need to use ${JAVA_HOME} to find where Java is installed this week, causes systemd to think that the dæmon is named "env". Josh Smeaton knows how to rectify this unsatisfactory state of affairs.

A lot of this is down to some needless marshalling of environment variables into Java properties. Maybe, in the future, the people who write WSO2 will adjust their software so that it just reads the damn environment variables directly itself given that Java programs can. Maybe, in the future, people will decide to have a way to invoke Java that's always in the same place. In such a future, things become moderately more streamlined.

[Unit]
Description=WSO2 Carbon Server service
Wants=mysql.service
After=mysql.service

[Service]
User=wso2
WorkingDirectory=/usr/share/wso2
EnvironmentFile=-/etc/default/wso2
RestartForceExitStatus=121
ExecStart=/usr/bin/java \
-Xbootclasspath/a:${CARBON_XBOOTCLASSPATH} \
-Xms256m -Xmx1024m -XX:MaxPermSize=256m \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=${CARBON_HOME}/repository/logs/heap-dump.hprof \
$JAVA_OPTS \
-Dcom.sun.management.jmxremote \
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog \
-Djava.security.egd=file:/dev/./urandom \
-Dfile.encoding=UTF8 \
org.wso2.carbon.launcher.Main

[Install]
WantedBy=multi-user.target

But the length of a 31 lines long service unit file is less important than the fact that it actually starts the dæmon in a reasonable manner, makes proper use of systemd, and doesn't have a mess of doubled up Poor Man's Dæmon Supervisors.


© Copyright 2015 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.