Monitor all the things

Following up from the previous work gathering Temperature, I finally got around setting up some monitoring.

Thanks to the good write-ups by alfaexploit.com, namely Monitoring Jail Resources in FreeBSD with Prometheus and Prometheus/Grafana Monitoring System on FreeBSD.

There’s also a document by the FreeBSD Foundation, Monitoring and Trending with Prometheus, still reasonably up-to-date for a reader in 2024.

I’ve not really used Prometheus before, previously having worked on very old school Nagios-based systems, then push-based solutions like Graphite and Riemann, to finally very Enterprise-focused tools like NewRelic. It’s fair to say I wasn’t a big fan of pull-based solutions because, on the old days, they incurred a cost in keeping a central database up-to-date and there was always a new host that someone would forget to add to the configuration.

A lot has changed since, and Prometheus is the de-facto standard these days.

On the other hand, I’m a big fan of Grafana, and their offering continues to be very solid. It was so quick and effortless setting up Grafana for the FreeBSD home server that it almost took the fun away!

I’m very satisfied with the quality of the offering and how easy it was to set up. At first I’m not sure where the Prometheus TSDB is being written. After digging thru the defaults of /usr/local/etc/rc.d/prometheus inside my monitor jail, I learnt about /var/db/prometheus.

I had to remember that this isn’t Graphite and Prometheus isn’t pre-allocating all the files and their retention ahead of time. Something I’ll have to figure out another time.

Finally, linking the Elitech RC4 data into Prometheus offered me two paths. Write to a text file and use the Textfile Collector or implement the Prometheus Python Client to link with the Elitech Python library mentioned on the blog post I first mentioned.

At a very basic level, this is the code that works:

from prometheus_client import start_http_server, Summary, Counter, Gauge
import elitech
from datetime import datetime
from time import sleep

TEMPERATURE = Gauge("temperature", "Temperature at probe tip, in Celsius")
HUMIDITY = Gauge("humidity", "Humidity at probe tip, in percentage points")
RECORDS = Summary("records", "How many records the RC-4 unit currently holds")
TIME_DELTA = Summary("time_delta_s", "How many seconds between RC-4 capture and our read")

def main() -> None:
    start_http_server(9430)
    device = elitech.Device("/dev/cuaU0")
    devinfo = device.get_devinfo()
    while True:
        latest = device.get_latest()
        RECORDS.observe(latest[0])
        TIME_DELTA.observe((datetime.now() - latest[1]).seconds)
        TEMPERATURE.set(latest[2])
        HUMIDITY.set(latest[3])
        sleep(devinfo.rec_interval.minute * 60 + devinfo.rec_interval.second)

if __name__ == "__main__":
    main()