Z Space Blog - Linux

Holiday 2023 Outage Post-Mortem

2024-02-02T00:00:00-08:00

In the days leading up to the end of 2023, zlg.space's primary hard drive began having errors and locking up the entire server. It began with simple unresponsiveness, and ended in a replacement drive, new OS, and a bumpy road of /opt self-compiled madness to bring the backup to the modern ages of PHP. The server is now on bigger, faster storage and an OS that won't get in the way if/when it breaks again, but it didn't come without an arduous test of patience.

December 28th, 2023

The problem happens for the third time since before Christmas. By the end of the year, I discovered via smartctl that the drive had experienced many hundreds of error events concerning the reallocation of damaged sectors. It would detect bad sectors, but either would not or could not tag them as bad and reallocate the data elsewhere. The hard drive was in a laptop-sized enclosure and had not been thrown, dropped, stepped on, or otherwise physically disturbed since October. At this point I determined it was some sort of unrecoverable hard drive damage. Before using SMART, I had tried to use fsck to repair filesystem errors, in case the problem was at that level. It didn't seem to help.

January 1st, 2024

I decided to order a new storage device in the same form factor; the enclosure's controller could correctly read and interact with other drives that I inserted, so I know it wasn't the enclosure which errored. This new drive is an SSD, so even if I put it into the same environment, it will be less prone to vibrations.

January 9th, 2024

The drive arrives and I start considering my options for moving forward. The primary candidates were Gentoo, KISS, or PiLFS.

January 11th, 2024

I tried one last time to get the drive to work, so I could attempt to get data off of it. I had performed a backup of the website's data a month or two prior, so not much data, if any, was lost. I guess there are perks to being the only active user of your own site, haha.

The server was formerly using Ubuntu Server, which uses systemd and cloud-init to provision your server as if it were part of a fleet. This is entirely overkill for a singular Raspberry Pi server, and includes many more layers of complexity than I need. I spent more time trying to figure out the Ubuntu way than I did just using tools I already know. "Modern" Linux with its container app formats and cloud-init shit and XDG portals and incomplete Wayland specs are just annoying. Like it or not, those technologies are not the way forward for Linux.

After reconfiguring the Pi's firmware to boot directly from USB, I chose to go back to Gentoo Linux for my server. I was leaning toward KISS Linux, but their Pi ecosystem isn't quite ready for what I need. Gentoo gets out of the way, has the tooling to pick and choose, and ebuilds are hackable for anyone who knows shell scripting. I really wish distros could get the experience right for developers, server owners, power-users, etc.

January 12th, 2024

Configuration and installation of Gentoo continued, and tweaks were put in place to reduce the number of writes that will be made to the SSD, to extend its life a little. Using a source-based distro is already going to be a mild problem when it comes to writes, but it can be mitigated.

January 13th, 2024

I fought against a chicken-and-egg problem created by the proprietary nature of the Raspberry Pi's Wi-Fi and Bluetooth, so I had to fetch distfiles for a few packages to bootstrap wifi. I could have just finished prepping ssh instead, but I want this machine to be adaptable to networking conditions. Also, who doesn't like watching the first boot?

The data from the backup was only from /srv, the Nextcloud installation, and the Postgres database cluster. System configs were not backed up before the old drive failed, but a backup from 2021 should make most of the restoration a smooth process. Now that this is all on one device, making backups should be trivial in the future.

Networking eventually was put into place, and after adding a few extras to get started, I now have a user accessible via SSH and the correct security needed for the admin account. We can begin the actual rebuilding!

January 14th, 2024

The real Gentoo Linux starts here. It feels wonderful to have a system ready to be hacked on! The process begins to put together a firewall, webserver, smarter log rotation, and install the software I was running before!

fail2ban and logrotate are the first goals, which went off without any major problems. Mostly just small configuration and boot-persistence kind of stuff. The next major goals are lighttpd and certbot, which will open up most of this server's available software. I sure miss my Nextcloud...

January 15th, 2024

I had to work and handle some errands and chores today, so not as much time to do big things. I installed ddclient and got started on Gopher. Fun little story there. I went to install my Gopher server of choice, sgopherd, and learned that its only available version in Portage was out of date. "Hmm, weird. I guess I'll go ahead and update it," I thought. So I did what one does when you need to bump an ebuild, double-checked on current dev tools/standards, made sure things were good, and...

I'm aware it's in maintainer-needed status, but if I'm the only person who cares about it in five years, I'm not bothering to interface with the community. I don't mind maintaining my own ebuilds, having prior active experience with the distribution. I have no plans to publish a repository/overlay, however. My prior experiences weren't a net-positive, so I will not be sharing my work. People don't think it be like it is, but it do.

January 18th, 2024

It didn't take overnight; instead it got hung up on dev-lang/rust-bin due to insufficient space in /var/tmp/portage. No worries, I updated the package.env file to use the SSD instead of RAM, cleared $TMPDIR, and moved on. It wasn't going to take all night anyway.

Eventually, I orchestrated the dance of lighttpd, lexicon, my registrar's DNS servers, and certbot to get a new certificate through Let's Encrypt. Now I can hook it up to cron again and wash my hands of it. I'm glad that having a partial certificate chain, and enough time between issuances was enough to get me back on track and not need any extra configuration or tweaking.

Later in the evening, php-fpm was brought up and, with it, the wiki is now online! Next in line are cgit/gitolite and mumble. Then we'll only have one service left to bring up...

January 23rd, 2024

I did some admin configuration behind the scenes to formalize system structure. It was smaller than I thought it was, which is a sign that I'm doing things correctly.

Due to the way Gentoo packages webapps -- and my complete lack of need for virtual hosts and other crap their tooling does, like needing versioned directories -- I compiled cgit myself and just threw it in a folder somewhere.

To be fair, Gentoo generally packages things in a decent manner, so when I have the time and patience I may revisit webapp-config. For now, I just set the vhost provider to lighttpd so it stopped yelling at me about apache2.

January 25th, 2024

Cgit was installed and hooked up to Gitolite. I verified that I could push commits, and Cgit was displaying the 'About' page correct, after telling it to use my own cm2html which calls mistletoe instead of markdown.

January 26th, 2024

It felt good to make some progress with relatively little effort, so I restored the Postgres cluster from a .sql backup and verified the data was present and ready to be interacted with.

January 27th, 2024

The database restore made MantisBT work almost immediately, so that was encouraging!

The final task, Nextcloud, took a number of days before I succeeded. Nextcloud 23 supports PHP 7.3, 7.4, and 8.0. The Gentoo install was on 8.1, and none of the 7.x series was available. So, I needed a way to run Nextcloud juuuuust long enough to get to a Nextcloud version that supports PHP 8.1. So, I tried with a third party overlay to install PHP in an older SLOT, because Gentoo is cool and can do that. Sadly, that did not work.

I'm not shy on the command line, so the journey began to build OpenSSL, PHP, and APCu outside of the Portage tree (generally not recommended). I installed things to /opt, customized a few things, and... partial success? occ didn't quite work yet but a phpinfo(); page worked mostly fine. We're getting there!

Across the 28th through the 30th, I wrestled with dependencies and other things that were needed, until eventually the occ command worked and I was able to upgrade to Nextcloud 24, which supports PHP 8.1!

January 31st, 2024

The circus continues as I tried to mitigate each new tiny problem that came up. The funniest took me some digging. Across the 6 or 7 rebuilds of PHP (each taking at least two hours), I learned that Nextcloud stores passwords with different crypto functions, and missing any of those functions makes login break! Why this isn't covered by its dependencies, I'll never know. BUT, be sure to enable argon2 in PHP if your Nextcloud logins don't work! I ended up locking myself out and having to disable bruteforce detection for my local IP for a bit before I found and fixed the password issue.

In the end, all data was okay and the server is in better shape now than it was before the event. But, because Nextcloud itself took nearly a week to get working, that is the first piece of software I'm targeting for removal should I have to do this dance again. There's no excuse for something like a webapp to be that difficult to put together and maintain as an admin.

Conclusion

Ultimately, I could not determine the root cause of the SMART errors. The drive had operated for around 30,000 hours across 3.5 years. I'm not sure if that's a good run or not, for a dinky little laptop drive. Vibrations from loud speakers could create the physical damage needed to eventually render the drive useless, but without greater access to tooling and a means to test, I cannot form a confident conclusion.

A few days after I discovered the problem, the drive failed to spin up. I don't know what any next steps would be for data recovery, but I'm confident that if anyone lost files, it was me, and only files or changes since December 1st, 2023. I'll hold onto it until I get the gumption to take a hammer to it. If anyone has ideas on further inspection before I destroy it, hit me up.

We'll just go forward with what we have. I apologize for such a lengthy outage. I would have fixed it sooner if real life hadn't been so adamant at getting in the way of things that are important to me. Keeping Nextcloud upgraded will prevent me needing to manually compile things in the future. The website is here, and that's all that matters.

I'm available for hire if anyone needs an administrator or someone who isn't afraid to get their hands dirty in the command line.

Free Software is Politics

2022-04-21T00:00:00-07:00

It seems that the systemd debate has made its way to Alpine Linux, and naturally that gets people out of the woodwork to bang their drum about it again. The irony's not lost, either. It's really tiring to see people complain about politics, in a software movement that's entirely about the politics of software to begin with. Have you guys forgotten why we're here? Are you more open source instead of free software? That's still a political statement; one of practicality and reduced costs.

For those unfamiliar, Alpine Linux is a Linux distribution (NOT GNU/Linux) powered by busybox, musl libc, and OpenRC. By contrast, systemd is a project aiming to provide a "system layer" between userland and the kernel. It explicitly depends on a recent build system (meson) and generally only supports glibc and the Linux kernel. It explicitly does not support any alternative libc. In short, Alpine and systemd go together like oil and water, on purely a technical basis. The social half even moreso, since this incompatibility resulted in users having access to a simpler system that isn't built on systemd. Choice is great, right? Some disagree, but they're free to make their own projects.

When dealing with the systemd debate, many try to reduce it to the early years of the debate, where people used the false dichotomy of technical and social. Or they derail and assume it's a personal vendetta against Lennart Poettering, without accounting for Lennart's behavior when he was advocating for systemd, and the smug ways that his project has treated contributions, criticisms, and distros like Gentoo. We do personal responsibility, first impressions, and all that civil shit, right? Can we not apply this to our fellow developers? Are we not responsible, to some degree, for how others perceive us and our projects? If so, then I think it's fair to criticize a developer.

Lennart said "Gentoo, this is your wakeup call," concerning the ability to run udev without systemd, when eudev was created. He actively asked GNOME to depend on systemd. Granted, GNOME was probably going to use it anyway due to Red Hat employing both Poettering and many GNOME developers... A number of Arch developers (particularly its sysvinit maintainer) were early contributors to systemd. It's safe to say, influencing other projects is the bread and butter of the systemd project, and I challenge anyone to argue otherwise.

If you're going to participate in the debate, at least understand the history of the project and the actions of the principal authors -- cultural origin matters here because their values are baked into the project. The discussions from the Arch and Debian communities are also worthwhile, to show examples of how NOT to deal with the issue. Arch promised sysvinit would remain as an option, then pulled the rug out from users with their switch. Debian had an ugly debate that resulted in some developers leaving the distribution altogether, and sparked the creation of the systemd-free Devuan distribution.

Arch and Debian both went systemd, while Fedora and SuSE leaned into systemd first. As a witness to both Arch and Debian's decisions as-it-happened, the one thing most distro devs missed was user transparency. Whatever plan you go with, you need a pathway for the users who get left out, or you're going to create a rift in the community or even a fork. So many users who didn't want systemd got stuck with it. Some people who were using systemd on neutral distros found themselves having to make a decision, where there wasn't a need to before. Systemd support was not uniform across the ecosystem (wrt build options), and still isn't.

The decisions affected a ton of users, ironically creating more forks and attempts to maintain control of one's computing -- the dreaded "fragmentation" boogeyman, often coming from those who don't see the value in a diverse software ecosystem. The fatal flaw of consolidation is the assumption that every developer in the problem space can agree on how its problems get solved. Given human nature, that is very flatly a false assumption. Every solution in software has different benefits and pitfalls. Some optimize for CPU speed. Some for memory. Some for I/O. Others for networking! No single project can solve every problem in a space. The solution to this is coexisting via multiple projects, all of which can solve the problem(s) in the style that they choose. The openness of software diversity is why systemd had a chance of adoption at all.

The ways that various distributions handled their decisions were decidedly high temperature, and it's clear from the beginning of the systemd project that it was intent on changing distributions. Denying that is to side with groups trying to manipulate the ecosystem into carrying their software on every default install. It's competitive behavior that has no place in a collaborative environment.

Not every piece of disruptive software causes this much trouble, though. One of Lennart's other projects, PulseAudio, hasn't caused nearly as much trouble even though it, too, has its share of critics. I posit that the limited scope of the project and its less aggressive appeal to the ecosystem allowed it to coexist with ALSA and JACK in a better way than systemd does with other inits. Many environments can treat PulseAudio as "another supported stack" with a little maintenance, without upsetting their software's architecture.

Even the Wayland project, with its own brand of smugness and the specific goal of replacing X11, hasn't done much to create an actual, damaging rift in the communities they are near. Programs can support both Xorg and Wayland; at least for the features that they both share.

It could also just be the nature of init systems. You generally have to make an all-or-nothing decision. Coexisting init or service systems are a somewhat new idea, which is why most distributions choose a single init. Apparently it's possible, though, to mix and match between OpenRC, s6, and runit. Neat.

The technical and social are inescapably intertwined, just like politics is to free software. Two sides of the same coin. The movement itself began as the desire to fix a printer driver and compute their way. Furthermore, the decision of which packages to include in a distribution is also political. It affects the "computing lives", for lack of a better phrase, of the users who trust the distribution's choices on software, or are otherwise aligned with their software philosophy. Anything that has far-reaching consequences on how people interact or conduct themselves could be interpreted as political.

So, what's really being called for when we're asked to "stahp being political"? To ignore the social history of a project? To censor jokes we don't find funny? To ignore prior debates, the outcomes of said debates, or the project management of upstream? Software isn't developed in a vacuum, and software with a big picture agenda (anything involving the desire to standardize is automatically political) is not doing itself any favors by accusing others of being problematic, when the politics is the point. It comes off as intellectually dishonest or in bad faith.

Why shouldn't we consider the whole when we make software decisions? If given the choice between two technically identical upstreams, would you rather deal with the upstream who's helpful and receptive to contributions and doesn't mind if people don't use their software, or one that sees itself above you, decides the structure of your system without helpful or constructive feedback for contributions, wants every system you touch to run their software, and has a nasty habit of displacing other software? You sure that last part is really an oopsie? Am I "not allowed" to notice that part?

I know what I'd choose, and it doesn't have to be highly personal. It's a matter of one's own software standards. As a programmer, I find it shameful to push one's project in a way that creates a lot of discord in the community. I don't try to get people to use VGStash over Backloggery or their other favorite game manager, but I might bring it up in a conversation if someone doesn't sound sure of how they want to fix that problem, and it's relevant to the conversation. Even then, I'd rather know that I have users because they like the software, and not because I screamed about it from the digital rooftops or pressured people into using it. That's a false merit, if anything.

Coexisting as a software project is not particularly hard. I could not, in good conscience, act in the ways that Lennart has regarding systemd, even if it was my own project. While some of the responses he's gotten have been over the top (death threats? Crazy), I think it's entirely fair to apply my personal standard to my software choices, and the people who write them.

I think there are many other people in free software who use similar criteria to determine which software they use. This is especially important for software that you think you might contribute to! If you don't like the ideas of the author, or their tech choices, or their philosophy to software, those are valid reasons to avoid their software! It's also valid to tell others that's your reason. They don't have to agree with you.

But, I have to question the motives and values of a person who tells me to stop paying attention to the politics that happen from a project with a decidedly political aim. The technical and social parts of free software cannot be reliably separated, any more than a human being's identity can be separated between heart and mind. Separating the art from the artist only shows half the picture.

Focus on the whole, and make the right decision for your computing. No outside entity knows your needs better than you do.

-z

(Who can read the GNU Manifesto and not gather that it's political?)