The known problems with Lyris ListManager

You've come to this page as a result of a question similar to the following:

What are the known problems with Lyris ListManager?

This is the Frequently Given Answer to that question.

You can find lots of people independently giving these same answers and making the same observations about Lyris ListManager, including Pamela Greene (pointing out some of these defects), Tom Lane (pointing out some of these defects), and Mark Booth.

There are various defects in Lyris ListManager:

Each of these is a serious bug in either the design or the implementation of Lyris ListManager.

If you are looking for someone to host a discussion forum for you, avoid companies that use Lyris ListManager (and especially those that extol its virtues), giving them a very wide berth indeed. These gross deficiencies and outright errors in Lyris ListManager vastly outweigh any possible redeeming features that it may have. The people who will end up actually using your discussion forum will thank you.

Lyris ListManager overwrites message IDs when sending mail.

Mailing list softwares associate MTS delivery problem messages with subscriber mailboxes, so that they can automatically deal with defunct or incorrect mailbox names. Modern mailing list softwares do this by using Variable Envelope Return Paths, where the mailbox name is encoded into the envelope sender of the message, and thus into the envelope recipient of any bounce message.

Lyris ListManager doesn't do this. Instead of modifying the envelopes of in-transit messages, it modifies the contents. In the mail messages that it sends to discussion forum subscribers who have chosen to read the forum via electronic mail it replaces the actual message ID of the message with a message ID that it has constructed. This constructed message ID contains the subscriber details. No mail subscriber, therefore, sees the real message ID; and no two mail subscribers see the same message ID as each other, or as anyone else sees, on any given message.

The problem that this causes manifests itself when such subscribers reply to any messages that they receive. The visible symptom is that everyone else's threaded NUAs and MUAs will show reply messages, sent by any mail subscribers, at the top-level, and not (as they should be) beneath the messages that they are in fact replies to.

The reason for this is not that the MUA used by the mail subscriber does not correctly support threading, as is so often erroneously argued. The MUA correctly supports threading, and takes the message ID of the message that the subscriber is replying to and inserts it into the References: header of the reply message, exactly as it is supposed to. The MUA is not the locus of the problem.

The problem is a Garbage In Garbage Out problem: The message ID in the References: header of the reply message is garbage because Lyris ListManager has supplied a garbage message ID in the replied-to message in the first place. The message ID in the original message being replied to is the fake message ID that Lyris ListManager constructed and overwrote the message with, when it sent the message to the mail subscriber, rather than the real message ID of the message. Mail subscribers never see real message IDs at all. Since every such subscriber sees fake message IDs, personalized for them alone, it is thus impossible for everyone else's MUAs to correctly reconstruct the chain of messages and replies in order to display the discussion by thread. This is because the message IDs in the References: header of the reply are the IDs that only one subscriber will have seen, and thus will not match the message ID in the Message-ID: header of any message that will have been seen by anyone else.

Every reply from a mail subscriber, therefore, forms a break in the chain because of Lyris ListManager, and it is made impossible for everyone else reading the discussion to track what is being replied to using the normal, simple, mechanisms that their User Agents provide for doing so.

The Lyris documentation tries to play this down by mis-portraying it as merely a complaint that one might encounter from "some users". However, this is in fact a fundamental design flaw in Lyris ListManager. The design assumes that any bounce messages resulting from incorrect or defunct mailboxes will include the content of the message that bounced, so that Lyris ListManager can scan the bounce message for the quoted Message-ID: field from the original message and from it determine the problematic mailbox name. But a common ploy of the senders of unsolicited bulk mail is to also rely upon bounce messages including the content of the original, and arrange for their mails to be delivered to their targets as bounce messages, by sending it to deliberately invalid mailboxes and setting the target's mailbox as the envelope sender. The resulting bounce message sends the message to the intended target. As a result of this, the widespread trend amongst mail administrators is not to include the original message in bounce messages. This eliminates the very behaviour that the design of Lyris ListManager is relying upon, however.

Lyris ListManager overwrites message IDs when receiving.

RFC 1036 § 5 describes the propagation mechanism for news messages between servers, and how it prevents loops in the distribution graph from causing the duplication of messages by dropping incoming messages whose message IDs it has already seen and processed.

Lyris ListManager prevents this mechanism from working. It unconditionally replaces the message IDs of all incoming messages, sent by forum subscribers, with IDs that it itself assigns.

The problem that this causes is that if anyone is smart enough to have their own private local news server carrying the newsgroups that they read, managing news on Internet in the ordinary way that millions have managed it for decades, they cannot easily have a two-way link to Lyris ListManager, in order to both read and write the discussion form. The standard duplicate article elimination mechanism fails, because an article sent to Lyris ListManager with one message ID will return the other way with a different message ID, affixed by Lyris ListManager, appearing to be a new message rather than the original message coming back again. People using newsgroups in the standard way, therefore, have to be content with a one-way article feed, giving only read access to Lyris ListManager forums.

Lyris ListManager doesn't correctly support NNTP.

RFC 977 describes the protocol that is supposed to be used to send and retrieve news between news clients and news servers.

Lyris ListManager doesn't adhere to this specification, albeit that the situation has improved slightly and Lyris ListManager no longer crashes as it once did.

It used to be the case that Lyris ListManager didn't correctly support the NEWNEWS verb. (You may encounter an installation of an old version of Lyris ListManager where this is still the case.)

An NNTP client, be it a pull feed or a user agent, should be able to send "newnews forum date time" to Lyris ListManager to request a list of the new articles that have arrived in the discussion forum since the given date and time. The NNTP specification does not require that it have selected a group first. But with older versions of Lyris ListManager, if it hadn't, Lyris ListManager would fail. Rather than return a list of article message IDs, Lyris ListManager simply closed the connection to the client. The lack of even an error message was a strong indicator that Lyris ListManager actually crashed when a client did this, which is why the connection closed so unceremoniously.

However, things are still far from being correct. For example: Lyris ListManager still sends bare linefeeds in its response to the HELP command, in violation of RFC 977 § 2.4.1. RFC 977 § 3.3.1 states that the help text sent in response to a HELP command is a textual response, and § 2.4.1 states that lines in textual responses be each terminated by a carriage return plus a linefeed. Lyris ListManager does not do this, and terminates lines in the help text with bare linefeeds.

The problem that this causes is that some NNTP User Agents use the HELP command as the nearest thing that the NNTP protocol has to a NOOP command, periodically sending that command to the server when they are otherwise quiescent in order to keep the connection "alive". Unfortunately, the erroneous bare linefeeds in the response cause these NNTP User Agents problems in parsing the response. The upshot of this is that if the User Agent is instructed to keep NNTP connections alive, it hangs when talking to Lyris ListManager.

Lyris ListManager does not use locally unique article numbers.

Article numbers are intended to be unique only within a single newsgroup. The same article number can exist in two different newsgroups. To select an article by number, one has first to select a newsgroup to provide context. Almost every news server software has article numbers that are only locally unique with individual newsgroups. The exception is Lyris ListManager. With Lyris ListManager, article numbers are global across all newsgroups that the server carries.

The problem that this causes is that newsgroups appear to be incredibly sparsely populated if there are a lot of discussion fora carried on the one server (as is the case with Media3 - one hosting service that uses Lyris ListManager). The article numbers of two successive articles are very often separated by gaps, sometimes of hundreds or even thousands.

Any NNTP client, be it a pull feed or a user agent, that needs to scan the newsgroup by article number for articles that it has not yet seen, ends up spending large amounts of time doing so. With other news server softwares, such NNTP clients can skip long unbroken runs of article numbers that they already have, speeding up the process markedly. With Lyris ListManager, article numbers occur far more often singly or in pairs, and there are not the long unbroken runs of already-processed article numbers to be skipped. Instead, there are large ranges of article numbers, the gaps between the real articles, where the NNTP client has to check for articles. Re-scanning the newsgroup by article number is a lot slower with Lyris ListManager than it is with other news server softwares.

Moreover, any NNTP User Agent that reports the numbers of unread articles in discussion forums will report highly erroneous figures, because such calculations are initially based upon the difference between the last recorded highest article number and the current highest article number.


© Copyright 2001-2003 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.