Hello, World

Introduction

I have been thinking I should write more. Perhaps today is a good start.

Back to BSD

Moved my home server from CentOS Stream 8, as it is EOL in a month’s time, and installed FreeBSD 14-RELEASE. Fun times learning the ropes again, last time it was around 7.0 or 8.0. Hopefully, the recent years using Mac for work mean that the muscle memory for GNU tools isn’t going to get too much in the way!

I kept delaying this migration, mostly because of data movements I had to do, and not wanting to buy another hard drive. Having to move from XFS to UFS, as the drive is USB-connected, is going to be interesting. Hopefully, fingers crossed, the backups have worked.

BSD installation was a breeze. I really enjoy reading the FreeBSD Handbook, it’s a true inspiration for how I approach documentation at work.

Installed a stripe zero (single disk) ZFS-on-Root with GELI full disk encryption, got the basic setup in, then started pulling all the stops. I’ll go a bit further on this, as means to use as a documentation should I ever need to reinstall again…

Only thing that took me a few iterations to figure out was the sdhci_pci0-slot0 Controller Timeout issue at boot, which caused it take a couple minutes more than needed. Thankfully, a forum of sibling BSD had the answer. Adding hint.sdhci_pci.0.disabled="1" to /boot/device.hints made the problem go away, hopefully without breaking anything else in the way.

Installed IPFW. A huge improvement to how much I was suffering with firewall-cmd on CentOS.

Git it good

From pkg(8) installed gitolite, not before creating a new ZFS dataset. One command and it’s sorted, pretty neat if I must say! This should help on snapshot and backup, if I decide not to continue with my current tooling.

zfs create zroot/var/git

Then it’s a matter of configuring the home of the git user to be under that mount point and it’s ready to roll.

I did wonder whether gitolite should be installed on a jail or not, decided not to as otherwise would’ve caused me additional headache on exposing it to the Internet later on, unless I went the trouble of using different TCP port than the main host etc.

A few nice things from pkg(8) like glances having the py39- prefix. Really good to see the dependencies being called explicitly, making it easier as a system to eventually move to newer system-wide Python version. This OS-as-a-whole vision that the BSDs have was always a huge selling point for me.

Encrypted external disks

I’m still carefully reading Chapter 20 Storage to ensure I understand gpart(8), I’m still on the old days of fdisk. Looking at gpart list and seeing GPT labels gives me a lot more confidence than the old BSD ways of doing slices on partitions. :)

I was unable to find a good set of instructions to create a GELI-backed UFS disk for my USB hard drive. The closest I found was a forum post of 2020 which I follow somewhat closely:

  • confirm which USB device it is, check dmesg, driver is da(4) so gpart show da0 to confirm what was there.
  • I had it with LVM-backed LUKS-encrypted, which FreeBSD 14 can’t deal with. As I had my backups, destroyed it: gpart destroy -F /dev/da0
  • created a new GPT schema on that disk, gpart create -s gpt /dev/da0
  • add new freebsd-ufs partition with gpart add -t freebsd-ufs da0 - created da0p1
  • encrypt with geli init -s 4096 -l 256 /dev/da0p1 - it asks for the passphrase now.
    • it also created a backup on /var/backups/da0p1.eli, 512 bytes long, suggesting geli restore /var/backups/da0p1.eli /dev/da0p1 can be used to restore it.
  • attach the geli(8) device to our system with geli attach /dev/da0p1
  • I created the volume directly on the eli device with: newfs -tU /dev/da0p1.eli
  • mounted using mount(8) normally
  • I’ll later review how to put a file-based access for this GELI on the second user key slot as someone suggested

Serving local media using Plex

Now, whilst I copy the backups into the newly formatted disk, started doing Jails. No surprises here, the Handbook is very complete.

Installing plex let me into a rabbit hole reading about FreeBSD Graphits and its support. This message on pkg install plexmediaserver I followed to the letter. I haven’t seen any difference, to be honest…

If you have a supported Intel GPU, you can leverage hardware
accelerated encoding/decoding in Plex Media Server on FreeBSD 12.0+.

The requirements are as follows:

* Install multimedia/drm-kmod: e.g., pkg install drm-fbsd12.0-kmod

* Enable loading of kernel module on boot: sysrc kld_list+="i915kms"
** If Plex will run in a jail, you must load the module outside the jail!

* Load the kernel module now (although reboot is advised): kldload i915kms

* Add plex user to the video group: pw groupmod -n video -m plex

* For jails, make a devfs ruleset to expose /dev/dri/* devices.

e.g., /dev/devfs.rules on the host:

[plex_drm=10]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path 'dri*' unhide
add path 'dri/*' unhide
add path 'drm*' unhide
add path 'drm/*' unhide

* Enable the devfs ruleset for your jail. e.g., devfs_ruleset=10 in your
/etc/jail.conf or for iocage, iocage set devfs_ruleset="10"

Please refer to documentation for all other FreeBSD jail management
utilities.

* Make sure hardware transcoding is enabled in the server settings

Plex works just fine - great stuff! I do “feel” an initial lag when playing video when I haven’t done so in a while, but I’m seeing this could be that FreeBSD is allowing the external USB-mounted hard drive to sleep more often, which I suppose it’s a good thing.

Backups and the Linux emulation layer

Creating another jail for SpiderOak One, for backups, which is itself a Linux application. I followed the instructions of Chapter 12 Linux Binary Compatibility adjusting for it to run inside a jail(8).

zfs clone zroot/jails/templates/14.0-RELEASE@base zroot/jails/containers/spideroak
mkdir /jails/containers/spideroak/var/git
mkdir /jails/containers/spideroak/data
service jail start spideroak

The follow steps are from FreeBSD wiki LinuxJails plus this to get apt working.

# in the jail using jexec
cd /tmp
pkg install debootstrap
debootstrap bionic /compat/ubuntu
# for some reason I had to manually install some packages?
cd /var/cache/apt/archives/
dpkg -i *.deb
apt --fix-broken install

There were lots of little errors, apt is pretty much useless for me on this state. I couldn’t even install curl without having lots of other dependencies it couldn’t resolve.

For now, just downloaded SpiderOak One DEB file from their website and installed.

Started setup with

SpiderOakONE --setup=-

After it synchronised its database I realised the mount points from my data disks into the jail were mounted at the jail’s root, and SpiderOak is running inside a chroot inside a Jail. Turtles all the way down, etc?

So I changed the mount points on the jail config similar to:

mount += "/var/git             $path/compat/ubuntu/var/git   nullfs   ro  0  0";

After restarting the jail to pick up the correct mount point, added those folders into SpiderOak:

service jails start spideroak
jexec spideroak
chroot /compat/ubuntu bash
SpiderOakONE --include-dir=/data/photos

Then confirming it worked with a

SpiderOakONE --build

It then gives you a hint how much data needs backing up:

Scanning...
{'aborted': False}
40542 queue entries left
Building...

This operation took some time as it’s a few hundred GB of photos, I still have to figure out how to start this as a service. I’ve also noticed that other jails, like Plex, run as a non-root user, but perhaps backups are naturally privileged? I made my jails read-only from disk, just in case…

How does it look so far?

At this point, I’m reasonably satisfied with the setup. It’s certainly very efficient, I feel the explanation is quite obvious on the screen, and it holds your hand in a decent amount, not trying to do things for you.

I’m wondering all these settings, whether I should create an Ansible or Puppet script to configure all of them. Considering how slow moving these are, and how one-off they are, I rather write a blog and move on. :-)

Hugo and Jenkins

To make life easier, I’ve picked Hugo as tool for blogging. It’s convenient that I can write on the Mac and the publish on the FreeBSD server.

This works better if Hugo gets called every time I push a commit to the gitolite we configured above, and sends to some CI server. I’ve found interesting the Jenkins is a first-class citizen in FreeBSD as even their own CI runs it.

First, enabled Gitolite hooks using their documentation and some helpful guidance.

Then, created new Jenkins jail

zfs clone zroot/jails/templates/14.0-RELEASE@base zroot/jails/containers/jenkins
vim /etc/jail.conf.d/jenkins.conf

jenkins {
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
  exec.clean;
  mount.devfs;

  # HOSTNAME/PATH
  host.hostname = "${name}";
  path = "/jails/containers/${name}";

  # NETWORK
  ip4 = inherit;
  ip6 = inherit;
  interface = em0;

  # openjdk
  mount += "fdesc   /dev/fd         fdescfs         rw      0       0";
  mount += "proc    /proc           procfs          rw      0       0";
}

Then, jexec jenkins and set up using sysrc(8):

pkg install jenkins-lts
sysrc jenkins_enable="YES"
sysrc jenkins_args="--httpPort=8180"
service jenkins start

As I kept going in-and-out of terminals I had to learn pkg info -D jenkins-lts to see the information during package install. Again, great points on the completeness of documentation.

Once Jenkins started, the password was at /usr/local/jenkins/secrets/initialAdminPassword.

I clicked the “Install plugins that the community most uses” and immediately regretted, luckily I’ve done my setup on a ZFS and can just destroy and start over.

For now, continuing with what ever we got. Created a new Multi-Branch Pipeline and pushed the blog.

Gitolite tried to do the post-receive we configured earlier and failed. Turns out we do need a token so Jenkins knows we are a trusted party. Documentation didn’t explain how to create the said token but I had a hunch it would be under the /manage/configureSecurity/ page on Jenkins. A section Git plugin notifyCommit access tokens had a convenient Add new access token, which I prompted used and updated on gitolite.

Now, configuring the jobs in Jenkins meant it needed an SSH key. It’s 2024 but this blog still applies, so created an ed25519 and added as a Jenkins credential. How to add a Credential on Jenkins is labyrinthine, Manage Jenkins > Credentials > System > Global credentials > Add Credentials.

Done this, added it to the Job I created, and then realise there’s no git package installed on the controller. Not even thinking yet how to do agents on this Jenkins, that’s probably going to be left to another day.

For now, after a few battles with the Git Host Key authorisation, which meant changing on Manage Jenkins > Security > Git Host Key Verification Configuration to Accept first connection, I have a Jenkins job triggered on commit, and that runs a simple Jenkinsfile that should call hugo.

It wasn’t too long before I realise that pkg install hugo brought emulators/hugo, a Hu-Go! is a PC Engine (TurboGrafx 16) emulator, when I should’ve installed gohugo. Time to learn about pkg autoremove to remove all those dependencies the “wrong” hugo carried with him.

Now, my Jenkinsfile

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh 'git submodule init ; git submodule update'
                sh 'hugo'
            }
        }
    }
}

has done the job, it’s a matter of publishing it.

Nailing the church door

Talking about this blog post to a good friend, an angel whispered joys of a sysadmin - certainly the catchy name I’ve been looking for all these years. Update hugo.toml accordingly.

Ultimately, a way for others to commiserate my endless saga, I shall publish this using a web server. Cue to another jail to come online, running nginx and hosting the public/ page generated be hugo earlier on.

To make my life easier I just created a subdomain in my DreamHost account and added a new stage to the Jenkinsfile:

  stage('Publish') {
    steps {
      withCredentials([sshUserPrivateKey(credentialsId: "jenkins-gitolite", keyFileVariable: 'keyfile')]) {
        sh 'rsync -avz -e "ssh -i ${keyfile}" --delete public/ username@server.dreamhost.com:felipe.grazziot.in/'
      }
    }
  }

Et voila!