On 2021-01-06 17:37:20 +0100 (+0100), Ildiko Vancsa wrote:
There was a question about potentially missing emails on the starlingx-discuss ML recently.
I asked around and it turns out that we had a mailman outage. While this happened the mail server still received the emails so once the queue runner for the ML was started again all the emails were sent out. This means that no emails should be missing, they just appeared later on the ML than when they were originally sent.
If there are any emails that seem completely missing please reach out to the Opendev community to raise the issue.
Please let me know if there are any further questions regarding this.
To get into more specifics, live migrations in the provider donating resources to the OpenDev Collaboratory, on which the mailing lists are run, caused that server instance to hang at around 04:00 UTC on 2020-12-21 and required a reboot to bring it back into service. When it booted, the queue runners for all the mailing list sites started successfully except the one for lists.starlingx.io, which went unnoticed until Saul brought it to our attention in IRC at 20:30 UTC on Monday (2021-01-04). Clark quickly identified the problem and brought that queue runner daemon online within five minutes, at which point all backlogged posts from that two week period were delivered to list subscribers and recorded in the list archives. I counted something like 57 messages for starlingx-discuss (most of which were automated testing results) received in the minutes after service was restored. The backlog won't be apparent when looking at the list archives however, as the messages are recorded there with the original dates and times they were queued, rather than when they were ultimately delivered. -- Jeremy Stanley