Posts on joys of a sysadmin

rclone and ScaleWay Storage as backup strategy

Thu, 27 Mar 2025 23:00:00 +0000

Exploring a few changes to how, and where, I host my own data, I revisited my backup strategy.

For years I’ve been using SpiderOak ONE and my main complain was the pricing slide. At 500GB backed up, I can’t be at the 400GB plan, so end up forking over US$150 a year for a 2TB plan.

I probably don’t need those 500GB, as I know from their UI that it once backed up some useless data like logs and save games. The tool did a great job on data de-duplication, only counting once for every copy of the same file across multiple devices.

Monitor all the things

Sat, 17 Aug 2024 17:00:00 +0000

Following up from the previous work gathering Temperature, I finally got around setting up some monitoring.

Thanks to the good write-ups by alfaexploit.com, namely Monitoring Jail Resources in FreeBSD with Prometheus and Prometheus/Grafana Monitoring System on FreeBSD.

There’s also a document by the FreeBSD Foundation, Monitoring and Trending with Prometheus, still reasonably up-to-date for a reader in 2024.

I’ve not really used Prometheus before, previously having worked on very old school Nagios-based systems, then push-based solutions like Graphite and Riemann, to finally very Enterprise-focused tools like NewRelic. It’s fair to say I wasn’t a big fan of pull-based solutions because, on the old days, they incurred a cost in keeping a central database up-to-date and there was always a new host that someone would forget to add to the configuration.

Temperature rising

Sun, 21 Jul 2024 20:00:00 +0000

Summer is upon us, temperature is rising, and I’m curious how my little server cabinet is faring.

A few years ago I bought an Elitech RC-4 Temperature Data logger for keeping an eye at the temperature at the office I used to work at.

It’s a pretty good device, you can leave it recording by itself, battery-backed, then read the data using the USB connector.

There’s even a Python elitech-datareader unofficial library, which did the job pretty nicely.

The five dysfunctions of a team (book)

Sat, 13 Jul 2024 20:00:00 +0000

Just finished reading The five dysfunctions of a team by Patrick Lencioni.

At first I was worried that the fable of Kathryn, a new CEO working to get the leadership team of a fictional start-up, was going to take the entire 280 pages of the book. The fable was good to set the talking points and did not get in the way of the message, easing my worries.

At the fable concluded, the book had already explained a lot of how the framework, or method, for identifying and correcting these five dysfunctions. Even better as then the book went into a more in-depth, more formal explanation around how to identify, what’s the impact of letting the dysfunction take place, and suggestions for mitigating them.

Specs and selectors

Tue, 25 Jun 2024 20:00:00 +0000

Back to basics today, writing from scratch some templates. ConfigMap, Pod, Service, Ingress, stitching them together, seeing the kubectl patch on a selector or on a metadata getting injected and magic happening on the ingress controller.

I was writing documentation, so going to the basics is important for clarity. Unfortunately, flipping a selector on a Service to match a new set of Pods isn’t a long-term solution, as you still have to figure out when to get rid of the old set of Pods you are trying to replace, and I don’t think kubectl patch without a more encompassing solution is sustainable.

Priorities

Mon, 24 Jun 2024 20:00:00 +0000

A reminder to self that sticking to the topmost priority the entire day pays off.

Working on Operations roles naturally has us jumping around, today was a good day when all was going as expected, had the time and focus to get an item on the top line of the Kanban completed, which also meant closing several other pieces of work that were pending action.

Feels good!

Bash, bang!

Fri, 21 Jun 2024 20:00:00 +0000

Reviewing a vendor’s code, learnt about the use of exclamation mark (!) on a variable.

Bash 5 documentation states:

If the first character of parameter is an exclamation point (!), and parameter is not a nameref, it introduces a level of indirection. Bash uses the value formed by expanding the rest of parameter as the new parameter; this is then expanded and that value is used in the rest of the expansion, rather than the expansion of the original parameter. This is known as indirect expansion. (…)

Testing and failing

Wed, 19 Jun 2024 20:00:00 +0000

Today at the day job we found out that one of our tests failed to capture a situation. That doesn’t mean our tests failed, it means we learnt one more scenario to test.

It is pretty comforting situation to even have a test to improve, rather than allowing ourselves to fail the same way in the future.

Minecraft

Tue, 18 Jun 2024 20:00:00 +0000

Some time ago, my eldest and I started playing Minecraft together.

As you would expect from a sysadmin at heart, we soon had a server running on the Linux-based home server.

That went well but I wasn’t very happy running everything on my own user account on that server. Any backdoor on that server, maybe an admin command that allowed a remote shell, and an attacker would have same access as myself.

Simpler, wins

Fri, 14 Jun 2024 20:00:00 +0000

Writing some “Quality of Life” helpers today. Finance wanted a breakdown of usage, but data on monitoring database was recording only the id of each application, not their name or to which business they belonged to.

Started writing something that queried the database, soon realised it was going to be an uphill battle. Needing a library with a binary, making it boring to deploy on a Lambda, or needing to figure out database access if running from our centralised GitHub Actions runners, all sorts of complications.

Consistency

Mon, 10 Jun 2024 20:00:00 +0000

In Naming Conventions I’ve mentioned the downside of failing to have certain conventions, which ended up causing an outage.

Writing the Post Incident Review of said outage, the good point was consistency.

By using the same tools across our group, the team responding to an incident had fewer unknowns to be dabbling with. This made troubleshooting much easier, and allowed us to focus on the things we didn’t know, then quickly turn around from full (service) outage into back into business as quickly as the automation job could run.

Delete more

Sat, 08 Jun 2024 20:00:00 +0000

Watching a video about an useful Executive Functioning hack, I’ve decided to take more control and started clearing the inbox of all those newsletters that I had to read a link or other.

Poor people at my work’s Slack as I just bombarded them with links I wanted to share. :D

Two in particular made me want to write a few words.

Lessons Learnt from Twenty Years of Site Reliability Engineering

The article doesn’t have a date other than a mention the time before GMail existed, so the SRE function at Google is at least pre-April 2004.

Practical Doomsday

Sat, 08 Jun 2024 20:00:00 +0000

Finished reading Practical Doomsday by Michal Zalewski (lcamtuf).

Having previously enjoyed his Disaster planning for regular folks, this was a good reading.

Michal does a great job in setting the scene that everyone should have a least a minimum plan on what to do. Disasters can happen at any time, from a multiple day power outage, to record-shattering floods.

Some of the suggestions around planning reminded me of Christopher Burgess’s Red Folder, the idea being you have a red folder at home with all relevant data about yourself and members of close family, and practical information about the house, car, insurance, etc, should the need arise.

Naming conventions

Fri, 07 Jun 2024 18:00:00 +0000

Today’s outage was brought you by having the same API token replicated in several places. Of course someone forgot about that place that runs only once a week, on a Tuesday, on a job that doesn’t alert if it fails.

Naming things is hard, always been, always will be.

But, the name doesn’t matter.

Next time, ask yourself: maybe we already have another copy of this token somewhere else?

Then you won’t need to be visiting several places to update all copies. Because, truth be told, you won’t.

Burger Joint

Tue, 04 Jun 2024 20:00:00 +0000

As a Platform team at work, we split our offering in three tiers.

alpha
beta
prod

Yes, yes, naming things is hard, why do you ask?

Today we were discussing how broken can alpha be.

I think a burger joint analogy works here.

If prod is your main street venue, the one where you take your wife and kids. You want the food to taste good, to be consistent every time you go, and probably good value for money.

Fail slow to recover fast

Mon, 03 Jun 2024 20:00:00 +0000

Back to work after a relaxing long weekend.

Today, intercontinental active/active deployments for API traffic, naturally on top of HTTP.

The keywords there already hint that you have multiple load balancers, at least a pair. What happens after each load balancer, say, traffic entering Americas and being served by Europe back-end, is outside the scope.

You want to control how quickly the API clients stop hitting the Americas load balancer if that Region went down.

AI first time

Thu, 30 May 2024 20:00:00 +0000

Doing the work at core of what I planned to do this week took half day, lots of fun. Doing the scaffolding around the code so that it could be deployed and used, four times that. Not so fun. Not hard, just … a long of moving parts?

Anyway, it works as intended and I’m pretty happy.

Almost 6pm I realised I needed something and, following a recent conversation with a colleague, gave the day job’s AI helper a try. Asked it to help me with a for loop on Terraform. It did.

Jenkins, love and hate

Wed, 29 May 2024 20:00:00 +0000

A moment of love/hate with Jenkins today.

Hate first, as was reminded me how brittle it is as world around it moves forward. Reasonably sure there’s a core functionality that was moved to a plugin and I’ll have to figure out what that is.

But lots of love, this has been running (patched and updated…) for over four years now, and it still does its job.

From time to time I look back and think we could’ve done it so differently. I’ve now becoming more accustomed to realising that this view, the hindsight, makes us forget that many of the nice things we have now did not exist four years ago.

The past

Tue, 28 May 2024 20:00:00 +0000

Interesting day to try and put a positive spin on things.

Looking at legacy and reminding oneself that things that have been running “like this” for a long time are a good thing. Means they are working as intended.

Is there always a way to make it better? Possibly. But nothing beats “done, for what we needed right now”.

RYOMS

Mon, 27 May 2024 20:00:00 +0000

Felt good to sponsor Run Your Own Mail Server kickstarter.

I haven’t read mwl’s work before, other than the first chapters of Absolute FreeBSD as available for Prime readers.

I liked his style and have followed the writing of the book as he posted on Mastodon.

As someone that used to run mail servers professionally, it hasn’t (yet?) excited me enough to try and running my own. Maybe the book could change that? Hope not, mail servers are endless amount of pain. :D

Civ VI parenting milestone

Sun, 26 May 2024 20:00:00 +0000

Been playing computers with the kid for a while. Multiplayer games more recently. Copious amount of OpenTTD and Minecraft.

More recently, got them hooked on Civilization IV.

Today’s milestone was completing our first multiplayer co-op together, a nice Science victory to crown the day.

Try

Sat, 25 May 2024 20:00:00 +0000

Meeting with neighbours today, talking about children etc, and one said a mantra that resonated well: “Try”.

It might have not sufficed for Yoda but life is not a Hollywood movie.

Today’s little victory is reminding ourselves that “Trying” is a great step.

R53R Hub Probe

Fri, 24 May 2024 20:00:00 +0000

At the day job we use Amazon Route53 Resolvers for some DNS access from cloud environments back to the datacenter domain controllers.

These work fine but Amazon does not do a great job on monitoring. The only CloudWatch metrics they expose are about traffic reaching from the various EC2/ENI instances on the VPCs that want to go to the Route53 Resolver Outbound Endpoint, not whether the endpoint was successful in getting something back.

Seize the day

Thu, 23 May 2024 20:00:00 +0000

It might be cheesy but, cherish the little victories, every day.

My wife told me that, and I might do just that. Write a quick reminder of something nice that happened on each day, and why not write here?

Glass House

Thu, 28 Mar 2024 21:24:00 +0000

Like any other story, it all starts with lots of pamphleteering and a cost saving opportunity. Local FTTH broadband provider pushes pamphlets every other week for months, enough to have you thinking… why shouldn’t I?

Your contract with the big telco selling Internet and phone to you is reaching an end, but inertia means you are very likely to just renew. By the last week or so, you call them for renew as the website doesn’t show you any renew options. Something is wrong in their system, you can’t renew. Their suggestion? Cancel, sign-up again, and by the way your telephone number will change.

Hello, World

Sat, 02 Mar 2024 20:14:41 +0000

Introduction

I have been thinking I should write more. Perhaps today is a good start.

Back to BSD

Moved my home server from CentOS Stream 8, as it is EOL in a month’s time, and installed FreeBSD 14-RELEASE. Fun times learning the ropes again, last time it was around 7.0 or 8.0. Hopefully, the recent years using Mac for work mean that the muscle memory for GNU tools isn’t going to get too much in the way!