I have recently updated a server of mine from Ubuntu 16.04 LTS to 18.04 LTS.
As usual, it was a bumpy ride. The upgrade stopped half-way through with my
system in a stupid state, and I had to figure out how to fix things so that
apt would be able to continue the process. I don't understand why this keeps
happening when the resolutions are so daft. This time it amounted to removing
a good deal of packages with dpkg. I don't understand how it is valuable
that I do this myself – it's not as if I know what I am doing.
However, in the end I was able to upgrade my system – yay! – but there were still some things to deal with, most notably my node.js services and utilities. Node.js is rapidly climbing to the top of my list of least-favourite technologies because things that used to work simply stop working. It just falls apart.
I discovered npm rebuild which rebuilds native libraries inside
node_modules without fetching those modules from the Internet again. This is
extremely welcome to ward off any stupid inconsistency problems with npm
package resolving. And it did solve my immediate node.js problems. But there
was a problem lingering still, which I thought was the fault of node.js or
npm, but turned out in the end to be systemd.
I have been using systemd for running storage-server, a personal project, on
my box with socket activation. The .socket-file looks like this (it sets up
listening on a Unix domain socket, which nginx connects to):
[Socket]
ListenStream=/run/storage-server.socket
[Install]
WantedBy=sockets.target
and the .service-file used to look like this:
[Unit]
Description=Storage Server
[Service]
ExecStart=/usr/bin/node /opt/storage-server/index.js
This used to set up socket activation perfectly fine, but the
systemd-library had now started throwing error messages that "No or
too many" sockets were available. An unnecessarily and annoyingly vague error
message. I discovered that I was able to debug socket activation with the
systemd-socket-activate command line utility:
systemd-socket-activate -l 127.0.0.1:8081 /usr/bin/node ./index.js
This worked like a charm, so the library, and indeed node.js, was off the hook!
In the end it turned out that systemd now requires a Requires declaration in
the .service-file to get socket activation up and running, making my
.service-file look more like this:
[Unit]
Description=Storage Server
Requires=storage-server.socket
[Service]
ExecStart=/usr/bin/node /opt/storage-server/index.js
This addition made the service run like before again.
