tl;dr: I built a cheaper, more reliable and private home internet connection using a Linux box, a VPS, and cellular modems. Cellular links are variable: peak-hour contention brings low bandwidth, jitter, packet loss, and brief outages. This post explores the problem and several ways to improve it.

Building reliable cellular broadband using VPNs

Over the last while I have experimented with ways to make cellular connections more reliable for personal internet access. This looks simple at first, but quickly becomes exponentially more complicated as edge cases emerge. The project had three goals. First, I wanted reliability: a consistent connection quality, which is particularly important when the network is highly contended at peak times. Secondly, I wanted to preserve some privacy. I do not trust UK consumer ISPs, and any news site will give a litany of cogent reasons why. Finally, the build had to be as cheap as possible. Everything ran on commodity hardware in a Unix environment. This page covers the 4G/5G modem load balance and cellular failover router approaches I tried, and records the experiments and ideas about how they could be developed further.

Rationale

Without fibre, copper, or Starlink broadband where I live, cellular is the only option for internet connectivity.

Cellular performance is highly variable, and connectivity is significantly worse during peak hours, particularly in the evenings. In general, expect low bandwidth, short periods of complete connectivity loss, high packet loss, and high jitter. The experience can be frustrating. The project started as a way of improving it, mostly as a fun experiment, with a meaningful improvement as a bonus. The primary aim was greater reliability, especially under peak-time contention. I trust UK consumer ISPs far less than reputable international hosting providers such as Hetzner, given their questionable histories and ethical standards, particularly in light of the Regulation of Investigatory Powers Act.¹ Hetzner hosts its services in countries with stronger privacy legislation. A further factor that increased performance variability was their traffic shaping and per-service QoS rules.² I wanted to reduce their impact. Therefore, an obvious solution emerged: encrypting the traffic with WireGuard or another VPN protocol.

In the UK, there are three MNOs (mobile network operators), VodafoneThree, O2 and EE.³ If you have three identical modems with three SIM cards, you can connect to three different networks, each with its own spectrum and infrastructure.⁴ Each connection will have different characteristics depending on factors such as your distance from a mast and local network conditions, including the number of users and their traffic. Cellular links are inherently contested.

If a provider’s mast fails because of a hardware, backhaul or power failure, it is completely outside my control. The providers may also share some hardware,⁵ but I think there is enough diversity between their networks for this not to be a major concern. By contrast, I can move my VPS between providers relatively easily, particularly because I manage the infrastructure as code. The least reliable parts of the system are therefore the radio access network, where contention is unpredictable and outside my control, and the operators’ infrastructure, including its hardware, backhaul and power, which is similarly beyond my control.

The internet has diverse routes, which provide considerable reliability. The first hop is the least diverse part of the route, so using multiple providers reduces the risk of a local outage affecting every connection. Multiple providers can also support an active-active link aggregation setup because each provider offers its own spectrum and bandwidth.⁴ This is why I used two modems, to improve reliability and increase bandwidth by using multiple parts of the spectrum, backed by separate infrastructure.

Conceptually, the goal was encrypted LACP. I tested everything as I went, often using the test setups as my primary internet connection.

The test setup

The test setup was two cellular modems connected to a Linux client. Each modem had its own SIM card and therefore its own IP address, and multiple addresses created strange issues; the goal was one point of presence on the internet. The plan was a layer 3 WireGuard connection to a cheap VPS, which would forward the traffic to the internet.

Terraform provisioned the cloud resources, and Ansible deployed the software to the Linux hosts. Most hosts ran Debian 12, my current default OS. Where a router OS was needed, or when the Linux network stack became too much trouble, OPNsense was used.

The only hardware purchased for the project was two ZTE modems, one 4G and one 5G. An Android phone in USB tethering mode occasionally served as a third modem for testing. As USB Ethernet devices, they all appear in Linux as enx* interfaces, which made identification easy. Everything else was hardware already on hand, such as old Lenovo ThinkCentre M53 mini PCs.

The setup was scrappy, built as quickly and cheaply as possible. I designed the high-level algorithms and used AI to automate as much of the tedious work as I could, which made prototyping much faster. The tools also pushed me towards Ansible and Terraform more than hacking by hand would have. This is because generated output is easy to test and verify repeatedly, and an LLM is good at patiently iterating.

Docker Compose handled much of the initial testing, using one container per node, connected via the bridge interfaces Docker calls networks. Network-condition emulation then added jitter, packet loss, and latency to mimic real-world conditions in the lab. This made it possible to anticipate problems in the real setup and isolate issues without the non-deterministic variability of a live environment.

To collect metrics from my test setups and experiments, I used the standard suite of network monitoring and troubleshooting tools, primarily MTR and other ICMP tools. Occasionally, I used Wireshark and tcpdump, especially when diagnosing why something wasn’t working. I also ran background monitoring to gather longitudinal data. This approach allowed me to understand what worked, what didn’t, and where the problems lay.

Approaches I tried

The core problem was failure detection. Most of the testing used ICMP as the link health probe because it was quick and easy. BFD would arguably have been better, offering higher-resolution and bidirectional detection. Both, however, struggle on cellular links: with high jitter, aggressive detection only makes the links flap more, while failing over too slowly drops packets or leaves traffic on a degraded link.

One experiment tried to sidestep this with an algorithm that broadly worked as follows:

1. Download some random bytes over HTTP for about 10 seconds on the link.
2. Read the length of the downloaded data, which essentially gives the download speed.
3. Sort by speed and save the list state.
4. Re-test at a shorter interval if the result changed significantly; otherwise re-test after a longer period.

This was not the exact algorithm used, but close to it.

Early testing used Ryu, an OpenFlow control-plane implementation I know well, for link control, but only as a few very quick tests. Ryu suits custom switching and routing setups well, though state management is difficult. The project was mostly about link state, VPNs, and failure detection, making the tool inappropriate. I set that approach aside for the experiments that follow.

Initial approaches: ICMP + ip route ideas

The following approaches use a simple ICMP probe to decide whether a link is “up”, then use routing to move traffic off a failed link.

Approach 1: ICMP liveness + load balancing

The first approach was a relatively naive active-active load balancer that spread outbound traffic over multiple physical uplinks behind a single WireGuard tunnel to a VPS, which forwarded the traffic to the internet. The core idea was to route everything into wg0, then use fwmark to steer the encrypted packets out over individual physical links. A control loop pinged 1.1.1.1 with ICMP to tell whether a link was up or down and adjusted the fwmark priority when a link failed. That design is inherently asymmetric, covering only client-side load balancing. The same could possibly have been done on the VPS, though it would mean modifying the source and destination IPs. Perhaps this is where OpenFlow could have come into play?

Approach 2: OPNsense hacking

Would a different operating system prove more reliable? The question prompted an OPNsense version of the same idea, which also swapped the Linux network stack for the BSD one.

The configuration was deliberately minimal: a single LAN and a single WAN, with the modem as the WAN connection. ICMP detected link health, and an external Python control loop scraped the OPNsense web UI to modify the WAN configuration. The scraping turned out to be easier than driving the real OPNsense API.

The approach had two inevitable weaknesses. The failover relied on web scraping, which is inherently brittle, and it still relied on ICMP. It was an interesting experiment and a good proof of concept, but not a credible route to production, even in a homelab context.

Approach 3: A more refined approach

The third test combined the two previous ones and went back to Debian. OPNsense had offered little benefit, and its control plane added another layer of difficulty. The idea was to set up two WireGuard tunnels to the VPS, pin each tunnel to a modem, and send traffic down one of them according to link health.

To force traffic down each of the two paths, I used fwmark and set link priorities through it. The configuration looked something like this:

ip rule add fwmark 0xa table 10
ip rule add fwmark 0xb table 20
ip route replace default scope global nexthop dev wg0 weight 1 nexthop dev wg1 weight 1

That created a bootstrapping problem. Detecting link health, or even establishing the first WireGuard sessions, requires an initial routing configuration; only then can the rules be adjusted to the discovered link health. In practice the solution was two control loops, called auto-isp and auto-switch. auto-isp watches and manages the two modems; auto-switch updates the routing priorities from the measured link health. The diagrams below show how they worked.

auto-isp

auto-switch

To make it a full-blown router, DHCP and DNS servers could also run on the box. With this set, the Debian box had the same core features as a standard consumer router. The approach worked fairly well apart from the bootstrapping faff, but it still suffered from the same issues as the earlier ones.

The main weakness was the link health algorithm. Asymmetric ICMP link health testing is unreliable: even at 500 ms, one unlucky packet at peak time can mislead it. The USB modems also sometimes dropped off the bus. I think this was a hardware issue with my M53 PC, which is why auto-isp was needed.

Two improvements suggest themselves. Run the link health algorithm on better hardware, at both the VPS and locally, and watch the kernel’s TX/RX byte counters (/sys/class/net/<iface>/statistics/, tx_bytes and rx_bytes) rather than relying on ICMP. Byte counters show whether a link is truly passing data and should give a more robust signal. Perhaps that is a future version I could build.

Approach 4: Spray

The previous approaches all depended on complex integrations between several tools. Perhaps a single tool that integrated the required functionality might be better. My idea was to build a userspace program in Go with no external libraries. It would create a point-to-point UDP VPN that sends encrypted packets over multiple interfaces. Each packet goes out over every interface; the receiver keeps the first copy to arrive and discards the rest. In order, my priorities were: security, core functionality, simplicity, minimum jitter, low latency. The protocol is called Spray, because it sprays packets at a virtual private server.

The full draft specification is reproduced in the appendix. In summary, Spray works like this.

Spray is a point-to-point UDP VPN. Reliability comes from redundancy, each packet is sent over every available physical interface, and the receiver keeps the first copy to arrive and discards the rest. A single send counter and a single receive counter are shared across all interfaces, so a sliding window of the last 16 packets lets the receiver spot duplicates and reorder packets that arrive out of sequence. The default MTU is 1,400 bytes, leaving room for the Spray header and, when needed, a fragmentation extension keyed by fragment ID; fragments older than five seconds are discarded.

Liveness is handled with keep-alives. If nothing has been sent to a peer for one second, a ping carrying a Unix timestamp goes out. The peer answers with a pong carrying both timestamps, from which the round-trip time can be logged. A peer that does not answer within 30 seconds is treated as dead, and the ping rate rises to every 100 ms.

Authentication uses a pre-shared key. A challenge–response handshake exchanges two random challenges and HMAC-SHA256 answers, and both sides derive the session key as SHA256(PSK || initiator challenge || responder challenge). Data packets are only processed after authentication succeeds. Payloads are encrypted with AES256-GCM; the nonce combines the 16-bit session nonce, the 16-bit counter, and a random 64-bit value, which keeps it unique across counter resets. The counter itself is reset by exchanging a new nonce when it nears its limit, with the old nonce kept for five seconds so in-flight packets are not lost.

The specification was handed to an AI to build a small prototype, which was then checked in a Docker Compose lab. I used Wireshark to decode the packets and confirm that the structure was as intended, and ran the same tests against a real VPS and the cellular modems. This showed that the concept was viable.

Whilst the prototype worked, bringing it up to a standard suitable for publication or use would take considerable time. Security was my main concern as the protocol amounts to a custom encryption scheme, and I do not generally trust myself to write them. It would also need serious reliability testing. The experiment was interesting and has potential, but I shelved development for now. Perhaps one way to take this forward would be to replace my custom cryptography with WireGuard or another well-established encrypted packet protocol, then integrate it with Spray.

Problems I encountered

Several problems recurred across the experiments. This section describes them.

The first problem was hardware flakiness. More robust, high-quality hardware (perhaps enterprise-grade) would have avoided many of the strange failures. The USB modems were the main offenders, occasionally dropping off the bus for inexplicable reasons, probably a symptom of the cheap PCs hosting them. That acted as a confounding variable and required extra engineering to bring the modems back online and repair their configuration after each drop. One of the same modems has since run directly off a PC with a faster bus and better power delivery without dropping, which supports the diagnosis. How much it affected the results overall is unclear, but most of the trouble could be traced to the nature of cellular connections in contended areas rather than to the hardware.

Similarly, there was a virtual hardware problem with the VPS. Most VPSs have only a single interface, though some with more exist. Three public IP addresses are needed for my idea to work, one shared egress address and one ingress address per modem. On Linux, each public IP realistically needs its own network interface. The intended VPS arrangement was as follows:

Both of these issues have a couple of common fixes: either more hardware or more software. On the surface, the cheaper option is just to use the Linux networking stack to hack around it in software. The network stack becomes exponentially more difficult to build with and understand as the demands on it grow, which is arguably because it is a complex system. It also shows that software is not always the right fix. Splitting things across multiple kernels and therefore, generally, more hardware can simplify things. If you value your time, a little extra hardware is often the cheaper option.

The main technical challenge with this project was fast failure detection. Unfortunately, fast failure detection is hard, especially over a link with substantial jitter and a high number of dropped packets. Both ICMP and BFD need a fairly clean link (relatively low jitter and low loss) to work well. The asymmetric topology, with several outgoing interfaces at the client and a single interface at the VPS, made failure detection and routing harder still.

There are two further technical challenges. The first is my aim of aggregating the links to substantially increase bandwidth. This was exacerbated by my requirement that packets be encrypted. The aim of increasing bandwidth by using several modems at once, under encrypted tunnels, compounded all of this. Substantial bandwidth gains need flow hashing, which cannot be done below the tunnels. There are two ways around it. One is to integrate the two tightly, as Spray does. The other is to put a bridge network interface over the WireGuard tunnels and accept another layer of Linux configuration.

Conclusion and next steps

A simpler approach works reasonably well: a good-quality 5G modem on an ISP with strong coverage, behind a plain WireGuard VPN. Connecting the modem directly to the machine also improved reliability, mostly by reducing complexity.

Another way to develop these ideas would be to build a custom binary for use in monitoring solutions, using kernel byte counters for failure detection. As long as a link has sufficient throughput, this should not be difficult and would be a useful tool to have.

Several approaches could still improve reliability and bandwidth materially, a goal which looks attainable. The testing was probably too hacky to make daily use worthwhile, though a little more time might have changed that. Taking the work forward would require forcing more traffic through both links at once, for bandwidth and reliability. I would also need to accept the cost of more expensive (or more) VPSs, which I had been avoiding, but they would make symmetric failure detection much easier. I should also use better local hardware. The most logical next step would be to use BGP with BFD for balance, as it is probably the simplest approach. This work has shown the ever-present conundrum with this sort of project. Both LACP and WireGuard are commonly used technologies, but within these constraints they become a complex integration problem. On the one hand, working within artificial constraints is always very interesting and provides a clear learning opportunity. On the other hand, they increase the time taken and frequently reduce the utility of the project.

Appendix: Draft Spray Protocol Specification

1. Packet structure

A UDP Spray packet MUST have the following structure:

• All time MUST be specified in milliseconds, including Unix timestamps.

2. MTU handling

• The default MTU MUST be 1400. This caters to underlying infrastructure expectations and allows for the Spray header.
• When required, a fragmentation header extension MUST be appended after the base Spray header.

The fragmentation header extension MUST have the following fields:

• The reassembly buffer MUST hold fragments keyed by fragment ID.
• Fragments older than 5000ms (configurable) MUST be discarded.
• When all fragments have been received (more-fragments = 0 and no gaps), the packet MUST be reassembled and forwarded.

3. Counter resets

• The Spray header MUST include a 16-bit random nonce generated on startup.
• When the counter reaches 65000, the implementation MUST send a reset and generate a new nonce.
• When the program starts, it MUST send peers a reset packet with the following structure:
  • Type = 1 (reset)
  • 16-bit nonce
• If a reset is received with a new nonce, the implementation MUST reset the expected counter to 0 and store the new nonce.
• The old nonce and counter MUST be stored for 5000ms (configurable) to handle in-flight packets during the nonce transition.
• Packets arriving with the new nonce during the overlap period MUST also be handled.
• This overlap period allows for the sliding-window deduplication.

4. Keep-alives

• If no packet has been sent to a peer for 1,000ms by default (configurable), a Spray packet MUST be sent with the following structure:
  • Type = 2 (ping)
  • 64-bit Unix timestamp of the send time
• If a Spray ping is received, a pong MUST be sent with the following structure:
  • Type = 3 (pong)
  • 64-bit Unix timestamp contained in the ping
  • 64-bit Unix timestamp at which the ping was received
• When a pong is received, the difference between the two timestamps MUST be logged.
• If no pong is received within 30,000ms by default, the implementation MUST throw an error and MUST NOT bother sending packets seen. It SHOULD also increase the ping rate to 100ms by default.

5. Authentication

• Before any data packets are exchanged, peers MUST authenticate.
• Authentication MUST use a pre-shared key (PSK) provided via a CLI argument.

5.1 Handshake

The handshake MUST proceed as follows:

1. The initiator sends an authentication request with the following structure:
   • Type = 4 (auth_req)
   • 32-byte random challenge
2. The responder replies with an authentication response containing:
   • Type = 5 (auth_resp)
   • HMAC-SHA256(PSK, challenge || responder_nonce)
   • 32-byte responder challenge
3. The initiator replies with an authentication acknowledgement containing:
   • Type = 6 (auth_ack)
   • HMAC-SHA256(PSK, responder_challenge || initiator_nonce)

Both sides MUST derive the session key as:

SHA256(PSK || initiator_challenge || responder_challenge)

Data packets (type = 0) MUST only be processed after successful authentication. If authentication fails, the implementation MUST log an error and drop the connection attempt.

6. Encryption

• Symmetric AES256-GCM encryption MUST be used.
• All Spray packet bodies MUST be encrypted.
• The GCM nonce MUST be constructed as 96 bits consisting of:
  • 16-bit session nonce (from reset)
  • 16-bit counter
  • 64-bit fixed random value generated at session start
• This ensures nonce uniqueness even across counter resets.

7. Multi-interface aggregation and deduplication

The multiple interfaces are merged and aggregated to improve reliability. Counters MUST be used to provide deduplication.

• The same receive counter MUST be used across all interfaces.
• The same send counter MUST be used across all interfaces.
• A sliding window of the latest n packets MUST be kept for reordering.
• n MUST default to 16 and MUST be configurable.

References