Cover by Vecteezy original

We’re all networked

Networks are everywhere in the 21st century. Most home setups are probably composed of just an ISP-provided modem + WiFi access point combo, and it serves most needs just fine judging by how the third party modem market has virtually disappeared since the 2000s.

But I run a NAS, want to stream my music when I’m out and about, and want to match domain names to my dynamic IP so I don’t have to memorize IP addresses. Over the years, I’ve had people (even other software developers) ask me about networking in general, so I hope this post is helpful in demistifying topologies that are a tad more complex than your usual home network.

If you want to skip the intro to networking basics, feel free to jump ahead.

Typical home setups

Most ISPs these days operate with fiber-optics backends and supply you a “modem” (even if the term is a bit outdated these days) that couples an Optical Network Terminal with a router, wifi AP and a few Ethernet ports.

The usual setup is for this box to be connected to the fiber trunk that goes into your building from the ISP, and then you plug your set top box to a specific port, and use the remaining (and the WiFi network) for all other home devices. The “specific port” bit is because the 4-port Ethernet switch on the back of the device sets up particular VLAN IDs for the IPTV streams that it needs to access to deliver your TV package’s channels, or any movies you rent. VLAN means “Virtual LAN” and it’s a way to tag Ethernet packets and share a physical link, but fit many logical ones inside it. So even if you plug something on that specific port, if you’re not tuned into the particular ID of that VLAN, you don’t see that traffic (but still can access the internet in general).

But I digress a bit - ISPs give you a box, which translates light from fiber-optics into Ethernet packets your devices can send and receive. You usually get a dynamic IP, which means every time your modem restarts, you get a new IP address. This is a problem for naming, as we’ll see further ahead.

The ISP provided modem usually also has a few extra features like a firewall or the ability to forward ports to a device in your home network. These features are usually basic or cumbersome like needing a reboot to be applied and so on.

Concepts

Let’s go over the basics of networking, so that the more advanced setup I’ll describe makes more sense.

The most common protocol for computer networks is IP, and it’s structured in a way to allow intermediate level protocols to sit between it and your applications, the most common of these intermediate protocols being TCP and UDP. That’s why you sometimes see TCP/IP as the “correct” designation for it.

The most widespread version of IP is v4 even as v6, which was standardized in 1995, addresses a lot of its shortcomings starting with number of possible addresses. The rollout of v6 has been in progress for almost two decades now, but has met with some challenges. Your ISP is sure to be using IPv6 behind the scenes, and may even provide you with an IPv6 address, but there are still too many interoperability issues with v4 for it to be a smooth experience - so I’ll be focusing on IPv4.

Basics of IPv4

At the bare minimum, an IPv4 network interface needs the following data:

• address
• subnet mask
• router (or default gateway, or just gateway) address

Also part of the configuration usually is the DNS server(s) address, which is not related to IP but mandatory to resolve names to their IP addresses.

Address

Starting with address, an IPv4 address is 32-bits long, represented usually in 4 groups of 8 bits (0-255 each). Not all addresses are the same though, as some are reserved for special purposes. That’s why most home networks will have IPs like 192.168.34.2 or 10.0.0.4 and your localhost address is 127.0.0.1. Addresses starting with 10. and 192.168. (or any of the other reserved ranges) will not be sent out to the wider internet, as routers know not to forward them.

Subnet mask

The subnet mask is less straight-forward, and usually trips people up. Netmasks are used to divide IP ranges in more manageable chunks. A netmask of 255.255.255.0 is typical, and it means for an example interface with 192.168.0.1 as address, it can talk to addresses ranging from 192.168.0.2 to 192.168.0.254. This gives you 254 addresses, which is usually more than enough for a home setup. The netmask value is applied to every IP with a logical AND, which gives us the network prefix, and one’s complement which gives us the host identifier.

| Address         | 192.168.0.15    | 11000000.10101000.00000000.00001111 |
| Netmask         | 255.255.255.0   | 11111111.11111111.11111111.00000000 |
| Network prefix  | 192.168.0.0     | 11000000.10101000.00000000.00000000 |
| Host identifier | 0.0.0.15        | 00000000.00000000.00000000.00001111 |

In this case, the subnet leaves us 8 unset bits for the host identifier, which means we have (2^8)-2 hosts in the subnet (254). We need to remove two addresses, since one is the network address and the other the broadcast one that can’t be assigned to specific hosts due to IP protocol needs.

Instead of using 255.255.255.0 for the subnet mask, you can also use the number of bits in it with the network prefix like 192.168.0.0/24.

Here’s another example for a less common /22 netmask setting:

| Address         | 192.168.20.88   | 11000000.10101000.00010100.01011000 |
| Netmask         | 255.255.252.0   | 11111111.11111111.11111100.00000000 |
| Network prefix  | 192.168.20.0    | 11000000.10101000.00010100.00000000 |
| Host identifier | 0.0.0.88        | 00000000.00000000.00000000.01011000 |

This network has 1022 ((2^10)-2) possible hosts, starting at 192.168.20.1 and ending at 192.168.23.254.

Router (or gateway) address

In every IP network, there’s a gateway or default address, sometimes also called “router”, to which all addresses outside of the current subnet are sent to, for forwarding. So if your address is 192.168.1.34 when you ask for Google’s Public DNS server IP of 8.8.8.8, it will be sent to your gateway (usually at 192.168.1.1, but not required to be this particular address) in hopes of reaching this IP outside of your home network.

External and Internal IPs

As we’ve seen above, your home network will consist of IP addresses that are part of some reserved range like 192.168. or 10. but those are not the IP address you’ll be associated with online. Your ISP assigns your router an IP that is routeable in public, so a distinction arises between this IP and your local ones. Your ISP’s router will bridge that gap, and having it as the default gateway for your home network makes sure that outside IPs are reached, too.

That’s why sometimes multiple devices in your network might be banned by IP, since they all have the same external IP and your router just “masquerades” as multiple devices on your behalf. Related to this, there’s one more concept that’ll be relevant for understanding my setup, which is port forwarding.

Port Forwarding

Port Forwarding is only really important when you’re serving content from your computer(s). If you run a web server on your laptop and want the internet in general to be able to access it, you can’t just tell people your address is 10.0.4.8 since as we’ve seen above, this is a reserved private network address local to your home network which they won’t be able to reach. But what’s a port anyway?

Ports are a way to direct network traffic to a specific service on a device. There are 65536 (2^16) usable ports, numbered from 0 onwards. Usually (and for historical reasons) ports below 1024 are for the most common services and restricted by the operating system to only be used by higher privileged users. Higher than 1024 port numbers are so-called ephemeral ones and are used by the OS for connections your programs might need. Over time, the space below 1024 has become crowded, so the distinction is not as useful anymore.

Some well-known ports are 80 for HTTP and 443 for SSL/TLS encapsulated HTTP traffic, or 22 for SSH. Browsers for example will default to these when you use an address, but you can also specify custom ones as HTTP does not mandate a unique port (neither does SSH, for that matter).

So, if you want to send someone over to your local device’s web server, you need your router to forward a port to your device. What this means in practice is that your router will accept traffic on a port number (say, 80) and send those packets to your device, as if they came from someone else.

An example of a router forwarding port 80 to port 3000 on a laptop in the network

Port forwarding plus a firewall is 99% of what most people need on a router. From the network separation you can see that just by using the router, most connections like Samba file sharing for example are blocked since the ports on the local devices are not exposed to the broader internet, unless your router has UniversalPnP on by default, that allows devices to open ports by themselves - I usually turn it off.

My home network

Alright, with the basics out of the way, let’s go into the details of my home setup.

I have requested my ISP to set my modem to “bridge mode”, meaning that it’s just a dumb translator from fiber to Ethernet and does no packet mangling whatsoever on its own, allowing my to manage all forwarding rules myself. This means I can’t hook up a TV box to it, but that’s OK because my ISP has an Android app that I’ve installed in my set-top box, so I can still access all my TV channels.

Since it’s in bridge mode, all traffic from it to my ESXi server is in red in the diagram, as it’s potentially dangerous. That’s why I have a separate port for it in my network rack. The server running ESXi has three network interfaces, so the first one receives this “WAN” traffic, and once filtered it goes out from the “LAN” interface onto my home network proper. This “LAN” network also has some virtual members in the form of VMs inside ESXi.

There’s also a management interface, since ESXi doesn’t allow you to share an interface for this purpose with one allocated to a VM. This access is just to manage the server via its Web UI.

Software

ESXi is great for this, as managing the virtual network assignments is super easy. Here’s a screenshot of how the WAN interface is mapped to the OPNSense VM:

And here the mappings for the LAN interface:

This is super easy to visualize! The pihole and traffic-cop VMs are on the internal network, just as any other device connected to the server’s actual Ethernet socket for this interface. I love ESXi, and will sure miss it 😢 since VMWare was bought by Broadcom, free subscriptions are no more and regular ones went up in price significantly. I hear good things about Proxmox VE so that’s my next stop.

I use OPNSense to manage the firewall and port forwarding, and it does about 100 other things that I don’t need. I’d love for an easier option to exist, but the others are either similar (pfSense) or have other compromises I’m not comfortable with. Still, it’s easy enough to manage the few port forwarding rules I have in place:

The first rule is put in place by OPNSense itself to prevent you getting locked out of its UI, and the others are for HTTP(S) traffic, Syncthing discovery, SSH access to my self-hosted instance of Forgejo, and then a rule for BitTorrent downloading of Linux ISOs and the like - while not strictly necessary, it does speed up downloads. There’s also a disabled (for now) rule for Telnet access to my Amiga BBS. The cool thing about OPNSense is that you can set an “alias” as the target for a rule, and then change it with its API. In my case, all aliases (syno, traffic_cop, MiST) just refer to a single IP address.

The other thing I use OPNSense for is to have static IPs for all devices via DHCP, which is a way to match the device’s unique MAC (Medium Access Control) address to an IP, which makes things more predictable around the network. I also set the Pi-hole VM’s IP address as the DNS server that is advertised on DHCP leases, so all devices in my home network block ads automatically.

So that’s it for the OPNSense VM. The other VMs I run are the aforementioned Pi-hole for ad blocking plus local DNS records (so they can be resolved by name, like syno.name.com for example) and a Debian 12 VM that just runs nginx with a set of rules to forward domain names to specific devices on my network.

In the diagram above, you can see syno and qoob as servers, and what I wanted was for domain X to always go to syno and domain Y to always go to qoob. I tried using HAProxy and relayd plugins in OPNSense for it, but it was difficult as their UI was not well-suited for my use case. So I settled on nginx’s ngx_stream_ssl_preread_module which works for me just fine. This is the config I use with it:

stream {
  upstream https_syno {
    server 192.168.1.27:443;
  }

  upstream https_qoob {
    server 192.168.1.73:443;
  }

  map $ssl_preread_protocol $upstream {
    default $name;
    "TLSv1.3" $name;
    "TLSv1.2" $name;
    "TLSv1.1" $name;
    "TLSv1" $name;
  }

  map $ssl_preread_server_name $name {
    hostnames;

    *.syno.domain.com https_syno;
    syno.domain.com https_syno;

    *.qoob.domain.xyz https_qoob;
    qoob.domain.xyz https_qoob;

    default https_syno;
  }

  server {
    listen 443;
    proxy_pass $upstream;
    ssl_preread on;
  }
}

With this simple config, I can point all HTTP(S) to this VM’s IP address, and it will reroute said traffic to the right device in my network. Crucially, it does not do TLS termination, so I don’t have to keep copies of the Let’s Encrypt certificates on this VM and can just manage them on the devices themselves!

Hardware

The ESXi server is an inexpensive Beelink EQ12 that rocks an Intel N100 4-core, 4-thread CPU with a 6W TDP, which means it has plenty of juice for the workload I have on it, and sips power which is important for an always-on server. It only has two NICs, so I added a USB-C to Gigabit adapter to use for the Management interface. I’m super happy with it! I got the 16GB RAM / 500GB storage option.

Syno is a Synology DiskStation DS1621+ with four 6TB drives and two 8TB ones, with 16GB of RAM and two 512GB NVME SSDs in RAID-1 for write caching. It has adequate performance with its Ryzen V1500B CPU, and even allows for link aggregation on its LAN interfaces (which I don’t use). I might upgrade it with a PCIe network card for 2.5G networking once I upgrade my switches to it. I run all (57) of my Docker containers on it, and it’s right now at the limit of what it can handle. The containers are all bunched up into a single docker-compose.yaml file.

Qoob is an off-the-shelf PC I assembled with an ITX motherboard and a Fractal Node 804 case. It was my old NAS, which I managed for years with OpenMediaVault but I got annoyed with some of their decisions and went for a turnkey solution instead. I will repurpose it in the future as a pure-compute node with no storage and maybe move my containers to it, and keep the NAS as just storage. I’ll need a beefier motherboard, as I’m currently rocking an Athlon 200GE with 16GB of RAM.

The mesh Wifi system is a mix of three Deco X20s and an extra M4 to cover my two-floor home. I used to have separate Asus and Netgear APs, but separate Wifis are a bit of pain, so this allows me to have a single WiFi network for the whole house (and separate IoT and Guest ones, too) with great performance. I have them all hooked up to Ethernet for backhaul.

Domain names

I use Gandi as my domain registrar, and have a custom quick and dirty Ruby script to update my records whenever my external IP changes (which is rare):

require 'faraday'
require 'json'

TOKEN = "REDACTED_GANDI_PERSONAL_ACCESS_TOKEN"
external_ip = `curl --silent http://ipecho.net/plain`

def update_record(record, external_ip)
  url = "https://dns.api.gandi.net/api/v5/domains/mydomain.com/records/#{record}/A"

  response = Faraday.put(url,
    {
      "rrset_values" => [external_ip],
      "rrset_ttl" => 300,
      "rrset_type" => "A",
      "rrset_name" => record
    }.to_json,
    {
      'Content-Type' => 'application/json',
      'Authorization' => "Bearer #{TOKEN}"
    }
  )

  raise unless response.success?
end

puts "external IP is #{external_ip}"
puts "getting current records..."

url = "https://dns.api.gandi.net/api/v5/domains/mydomain.com/records/*.qoob/A"

response = Faraday.get(url, {}, {
  'Accept' => 'application/json',
  'Authorization' => "Bearer #{TOKEN}"
})

response_json = JSON.parse(response.body, symbolize_names: true)

current_gandi_ip = response_json[:rrset_values][0]
puts "current Gandi IP is #{current_gandi_ip}"

if external_ip == current_gandi_ip
    puts "no update needed"
else
    puts "update needed!"

    ["*.qoob", "qoob", "@"].each do |record|
      print "\tupdating #{record}... "

      update_record(record, external_ip)

      puts "done!"
    end
end

puts "all done"

This allows me to access all of my services with for example outline.domain_i_own.com and not worry about when my ISP chooses to assign me another IP address.

Particular services

I won’t list all of the services as it’d take too long 😅 but here are some of the highlights:

• Outline a great Notion-like you can self host
• Gonic a Subsonic-compatible server I pair with DSub for listening to my music on 5G
• Grafana to monitor my Synology server and Shelly Plugs power usage numbers
• Jellyfin as a Plex replacement
• Linkding to store my bookmarks to read later
• Miniflux to follow RSS feeds
• n8n to have Telegram-triggered workflows (like send a bot I created a message with a video URL and have the server download it with yt-dlp)
• open-webui to interact with Ollama models running on my PC’s Radeon 6800 XT
• Plausible for this blog’s analytics
• Vaultwarden as a backend to Bitwarden’s mobile and browser apps

Parting words

My setup is not super complicated, and I left out how I use Tailscale to not expose SSH on the open internet, but the post is long enough already! Let me know if you have any questions or follow-ups on this toot.

Until next time! 🖖