Glass House

March 28, 2024

Like any other story, it all starts with lots of pamphleteering and a cost saving opportunity. Local FTTH broadband provider pushes pamphlets every other week for months, enough to have you thinking… why shouldn’t I?

Your contract with the big telco selling Internet and phone to you is reaching an end, but inertia means you are very likely to just renew. By the last week or so, you call them for renew as the website doesn’t show you any renew options. Something is wrong in their system, you can’t renew. Their suggestion? Cancel, sign-up again, and by the way your telephone number will change.

A butterfly flapping its wings. Call that FTTH provider, can you install? Yes? When? OK, get the clearance from landlord for yet another cable in the wall and let’s do this. In meanwhile, figure out how to port the landline number to a VoIP provider – I only use that phone for inbound calls, and that needs to work.

Installation happens, I don’t even need 900 Mbps, almost 20x my then very-acceptable-thank-you-very-much 50 Mbps. Static IP to skip the newcomer’s CGNAT network? Yes, why not? Will I miss the big telco’s IPv6 network? A bit, but HE Tunnelbroker still exists and latency to it wasn’t an obstacle. And by taking the routing and firewall to my own hands I can get the DNS Ad Blocking working to all cases, not just the IPv4 ones – as the big telco router wouldn’t allow me to keep IPv6 working without their own DHCPv6/SLAAC.

Then there was trouble.

The new ISP could provide their own router, which I opted-out as already have a working Wi-Fi Mesh. And having that home server running FreeBSD, I felt it was right time to promote from home server to gateway.

This write up is a testimony of how wrong I was.

Cutting to the chase, I’ve spent several hours reading https://docs.freebsd.org/en/books/handbook/firewalls/#firewalls-ipfw and trying to figure out how to get things working.

I wrote several ipfw rules that should be working but none did. Fast-forwarding to the solution, Chapter 33 of the FreeBSD Handbook explains how to configure your firewall, including NAT, but only Chapter 34 explains how to configure a gateway.

So, a firewall rule as simple as some blog posts explain, it won’t work until you set the computer as a gateway.

This, of course, is obvious in hindsight, especially for someone whose job was Network Engineer and have been messing with home-built gateways since Linux kernel 2.0.something. Perhaps the saying about not forgetting how to ride a bicycle forgets to mention you will fall a few times…

In a nutshell, make sure your /etc/rc.conf has:

firewall_enable="YES"
firewall_script="/etc/ipfw.sh"
firewall_logging="YES"
firewall_nat_enable="YES"

gateway_enable="YES"

you also need something on /etc/sysctl.conf to globally disable TCP Segmentation Offload as its optimisation on the network interface hides essential information needed by the NAT module on the kernel. Adding some log messages from ipfw when packets are denied also helps, the second line here:

net.inet.tcp.tso="0"
net.inet.ip.fw.verbose_limit=5

Lesson 1: even great manuals make leaps of faith.

Finally, a snip of the ipfw I was using, which sits /etc/ipfw.sh:

#!/bin/sh

WAN="em0"
LAN="ue0"
CMD="/sbin/ipfw"

$CMD -q flush

$CMD nat 1 config if $WAN reset

$CMD add 00005 allow ip from any to any via $LAN
$CMD add 00010 allow ip from any to any via lo0

$CMD add 00100 nat 1 ip4 from any to any in recv $WAN
$CMD add 00101 check-state

$CMD add 00120 skipto 01000 udp from any to any 53,443,500,1194,4500,51820 out xmit $WAN keep-state
$CMD add 00130 skipto 01000 tcp from any to any 22,53,80,443,1723,1701 out xmit $WAN setup keep-state
$CMD add 00140 skipto 01000 icmp from any to any out xmit $WAN keep-state

$CMD add 00200 allow icmp from me to any out xmit $WAN keep-state
$CMD add 00210 allow udp from 100.64.0.4 to any 67 in recv $WAN
$CMD add 00212 allow udp from me to 100.64.0.4 67 out xmit $WAN
$CMD add 00250 allow udp from me to any 43,53,123 out xmit $WAN  keep-state
$CMD add 00260 allow tcp from me to any 43,53,80,443,1080 out xmit $WAN keep-state

$CMD add 00299 deny log ip from any to any out xmit $WAN

$CMD add 00500 allow icmp6 from me to any out xmit gif0 keep-state
$CMD add 00550 allow udp from me to any 43,53,123 out xmit gif0 keep-state
$CMD add 00560 allow tcp from me to any 43,53,80,443 out xmit gif0 keep-state

$CMD add 00999 deny log ip from any to any

$CMD add 01000 nat 1 ip4 from any to any out xmit $WAN
$CMD add 01001 allow ip from any to any

I found it was pretty cool to have very restrictive firewall until it started to get on the way. VPN connecions, WhatsApp calls, all sort of random things would use other ports for their communication. I ended up with a simpler, less paranoid:

$CMD add 00120 skipto 01000 udp from 192.168.0.0/24 to any out xmit $WAN keep-state
$CMD add 00130 skipto 01000 tcp from 192.168.0.0/24 to any out xmit $WAN setup keep-state

Lesson 2: there is such thing as too secure.

I was also struggling with the physical networking - my good old mini-PC has a single Gigabit Ethernet. I need two ports to be able to be a proper gateway. Gigabit Ethernet USB dongle comes in from the big marketplace website, after reviewing it’s from a chipset recommended by the various blog posts. It’s recognised as ue0 with the axge driver, nice.

Performance with iperf3 ? Barely hitting 500 Mbps. Try on the native GbE port, full Gigabit. Try the dongle on the laptop? Full Gigabit, zero packet loss, beautiful.

I then learn about the various models of USB and their Generations. The miniPC is from 2016, so all it can do is USB 3 Gen 1, max 5 Gbps. It should be enough for a 1 Gbps Ethernet dongle but it’s recommended a USB 3 Gen 2 that can do 10 Gbps. Anyway, not sufficient for my use case.

Lesson 3: hardware support matters.

FreeBSD forums had several suggestions, and a bugzilla entry https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267514#c8 suggested approaches. Even with those, the interface was unstable, isc-dhcpd would start complaining, and suddenly packet loss would come and make it unbearable.

For a while I used an older 100 Mbps USB Ethernet dongle, which also happened to be with the same driver, and whilst it wasn’t as bad on the interface flap, it also had weird behaviour on the client nodes to my gateway. Pages wouldn’t fully render, links wouldn’t open, Android apps would report loss of connectivity randomly. Pretty bad.

One of the most curious finds in all of this was discovering I had to disable TXCSUM, the transmission checksum, on this 100 Mbps Ethernet USB to make it work.

I had done it all: ipfw configured, gateway set, and a few things worked from the clients: I could ping, but couldn’t browse or even resolve DNS. With tcpdump open on both gateway and client, I started seeing that the packets would exit the client, traverse the gateway, and the reply exit the gateway and get rejected at the client.

19:15:01.734919 IP (tos 0x0, ttl 64, id 16457, offset 0, flags [none], proto UDP (17), length 67)
    192.168.1.122.49746 > 9.9.9.9.53: [udp sum ok] 50934+ [1au] A? uol.com.br. ar: . OPT UDPsize=4096 (39)
	0x0000:  0060 6e00 0628 207b d2af 0b65 0800 4500  .`n..(.{...e..E.
	0x0010:  0043 4049 0000 4011 662d c0a8 017a 0909  .C@I..@.f-...z..
	0x0020:  0909 c252 0035 002f 1d95 c6f6 0120 0001  ...R.5./........
	0x0030:  0000 0000 0001 0375 6f6c 0363 6f6d 0262  .......uol.com.b
	0x0040:  7200 0001 0001 0000 2910 0000 0000 0000  r.......).......
	0x0050:  00                                       .
19:15:01.752406 IP (tos 0x0, ttl 59, id 43368, offset 0, flags [none], proto UDP (17), length 83)
    9.9.9.9.53 > 192.168.1.122.49746: [bad udp cksum 0xd484 -> 0xf0b3!] 50934 q: A? uol.com.br. 1/0/1 uol.com.br. A 200.147.35.149 ar: . OPT UDPsize=512 (55)
	0x0000:  207b d2af 0b65 0060 6e00 0628 0800 4500  .{...e.`n..(..E.
	0x0010:  0053 a968 0000 3b11 01fe 0909 0909 c0a8  .S.h..;.........
	0x0020:  017a 0035 c252 003f d484 c6f6 8180 0001  .z.5.R.?........
	0x0030:  0001 0000 0001 0375 6f6c 0363 6f6d 0262  .......uol.com.b
	0x0040:  7200 0001 0001 c00c 0001 0001 0000 0032  r..............2
	0x0050:  0004 c893 2395 0000 2902 0000 0000 0000  ....#...).......
	0x0060:  00

This bad udp cksum was the hint.

Lesson 4: USB Ethernet isn’t a real choice for a gateway.

I needed options. This mini-PC, doesn’t have expansion ports. Except, it has. An M.2 port, commonly used for SSD/NVMe disks, can also work as a PCIe expansion port.

Unknown to me, there are several keys to M.2, to ensure that the device is compatible with the motherboard. I discovered that the wrong way, after ordering an A+E M.2 for an M-keyed motherboard.

Return to seller, order a different M-key M.2 Ethernet. It arrives packaged on a bubble wrap, not on an electrostatic bag. Bad omen. Connect, no signal. Confirm I’m on latest BIOS, try to reseat, no dice.

At least I didn’t start making holes in the casing before I realised the network card was likely dead on arrival.

Lesson 4: vendors matter.

I’m not a hardware guy. I hate the possibility of damaging my own computers. I was not happy with the possibilities. Buying a different PC to act as home server and gateway was out of the options, budget-wise and also ecologically: the current NUC is working just fine!

I don’t recall how I ended up finding about the EdgeRouter. Less than £60 for something with several Ethernet interfaces, full-speed NAT with hardware acceleration, and it just works? I should’ve done that several days earlier.

Lesson 5: pick the right tool for the job.

I am happy with the setup now. The home server is back doing its job, the Wi-Fi Mesh doesn’t get in the way, and I can get the full 900 Mbps symmetrical even over the air.

An addendum why didn’t I use the Wi-Fi Mesh as router to begin with? Turns out that TPlink Deco, the solution I use for the Mesh, is pretty bad as router, no port forwarding options I could see, the LAN/WAN auto-sensing means my home server with statically assigned LAN IP was being considered WAN, all sort of little nags.

Finally, is my honeymoon with FreeBSD over?

Not entirely but Lesson 3 above was a big reality check.

On the same week I discovered that, about an year ago, Plex stopped supporting hardware transcode on FreeBSD> since version 1.32. This was certainly a bit of a bummer, especially as some of the suggestions are either to move to a different software or run Plex as a Linux guest on bhyve with a PCI pass-thru. Neither are great.

Lesson 6: inertia wins. Almost every time.

QED.