I recently attended EuroBSDCon 2024 in Dublin, where I spoke about end-system multihoming with ILNP. It was great to meet some new people and hear about other work being done with the BSD network stack -- and was also my first in-person international conference, which was a milestone for me!
The Multihoming Problem
Multihoming is the idea that a single machine could be connected to a network at multiple points via different links. This looks like a good idea, as the additional links would provide resiliency in the case of failure, and greater aggregate throughput. In practice, however, it is not so simple.
IP networks (such as the Internet) use addresses to identify nodes on a network, and to indicate their location. In order for a node to to have multiple points of connection, it must have multiple network locations, each of which requires a new address. However, different addresses also (incorrectly, in this case) imply a change in identity due to the overloaded semantics. In order for a communication session to take advantage of multihoming, some mechanism must be used to reconcile the set of location information with a singular node identity -- something that is not possible with IP addresses.
By way of analogy, consider postal addresses. My childhood home had a back gate opening onto a different street with a different name. We therefore had two different house numbers on two different streets. Mail could be sent to either address, which would frequently confuse postal workers as we had only one letter box -- which faced the opposite street from the address most frequently used!
One solution would be to embed this information in routing infrastructure. Continuing the analogy, this is equivalent to renaming one of the streets and updating the maps so that a single postal address could refer to multiple locations. While this simplifies the task of sending mail (I always use the same address), it makes life much harder for postal services and cartographers. At Internet scale, this translates directly into an algorithmic problem: every provider-independent address must be recorded separately, leading to rapid growth in routing tables when compared with traditional provider-aggregatable addresses. Shifting this complexity to the routing infrastructure is therefore undesirable.
The alternative is to push the complexity up the stack to applications -- this is the approach taken by QUIC. Rather than trying to fix the networking problem, application-layer solutions consider network-layer multihoming to be impossible, so instead create a set of single-homed connections to different addresses and use application-layer logic to multiplex traffic across them. This saves routing infrastructure, but instead requires developers of all applications to update their software -- a system-level solution that can operate behind the existing API abstractions is therefore preferable.
The New Solution
In this talk, I described an in-kernel solution that avoids pushing new complexity to routing tables or applications. This is possible with ILNP, which changes how endpoints understand the semantics of IPv6 address bytes, but does not change the API, or the syntax or behaviour of IPv6 packets when routing.
I also showcased some initial results obtained from round-the-world multihoming tests of our ILNP implementation for FreeBSD. ILNP interacts well with UDP, and dynamically load-balances traffic across all available paths even as the set of in-use addresses dynamically changes. TCP exhibits some stranger behaviour, as existing congestion control algorithms assume out-of-order delivery indicates loss or congestion. This assumption holds for single-path communication, but not for multipath, where out-of-order deliver is expected even without loss.
The source code for the FreeBSD implementation is available on GitHub, and my slides are available here. If the recording of my talk is release, I will link it here.