diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md new file mode 100644 index 0000000..4e72ee8 --- /dev/null +++ b/_posts/2019-08-19-awdl.md @@ -0,0 +1,227 @@ +--- +layout: post +title: "Meshing using Apple Wireless Device Link (AWDL)" +date: 2019-08-19 08:00:00 -0000 +author: Neil Alexander +--- + +### About AWDL + +I was mostly prompted to write this post in response to a [Hacker News +thread](https://news.ycombinator.com/item?id=20735462) recently, which announced +the release of an open-source implementation of the Apple Wireless Device Link +(AWDL) protocol. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay +and some other Apple wireless technologies. + +With a few exceptions, most wireless networks in the world operate in +"infrastructure mode" which is where a wireless access point serves one or more +wireless clients. Think of your Wi-Fi at home, at work or in a coffee shop. +However, as implied by the name, reliable and usable infrastructure Wi-Fi is +often only available in certain physical locations with "good infrastructure". +If you wanted to connect some devices together anywhere not served by an +infrastructure Wi-Fi network, or in a location where you can't suddenly plug in +a wireless access point, you may not have many options (Bluetooth aside). + +AWDL is designed to avoid this problem by extending the 802.11 wireless standard +to allow client devices to communicate directly with each other, without the +help of the central wireless access point. You can walk out into a field with a +couple of iPhones or Macs and they can use AWDL to discover each other and +exchange data, peer-to-peer. Even better is that nearby devices that are +connected to different infrastructure Wi-Fi networks can still communicate with +each other using AWDL! + +### The science + +Normally, when connected to a wireless access point, wireless clients remain +locked to the specific radio channel that the AP is using. AWDL works by +instructing the wireless adapter in the device to "hop" between channels so that +it can not only remain connected to the wireless access point, but can also +listen to other nearby devices. + +Devices announce their presence and information about their services on a +"social channel" for other devices to hear, effectively creating peer-to-peer +service discovery. Once two devices have decided that they want to communicate +directly, they agree to jump to another channel for real data exchange so that +they don't interrupt existing Wi-Fi networks or, indeed, the social channel. +These "hops" between wireless channels happen so quickly that there's very +little disruption to what the user is doing with their Wi-Fi connection already +(except for some minor wireless performance degradation - to be covered later). + +### Mesh opportunities + +Yggdrasil is designed to create a mesh network automatically out of +interconnected nodes - the idea being that all nodes can route to all other +nodes on the mesh network by routing through other nodes. + +Today, many of these connections happen between nodes across the Internet, since +the community is still relatively small and geographically dispersed. A node +joining the Yggdrasil network needs to only peer with a single device that is +already connected to the wider network in order to participate in the +fully-routable mesh. + +However, it's not the goal of Yggdrasil to remain something that we just toy +with over the Internet. We want to build a protocol that can scale globally and +work ad-hoc, even in places where infrastructure might not be particularly +strong otherwise. We think that one of Yggdrasil's greatest strengths is that it +is very close to zero-configuration, beyond giving it a very small number of +configuration options, and it should scale well too in principle. + +Yggdrasil can already discover potential peers on the same network segment by +using multicast service discovery, which sounds a lot like what AWDL does on the +social channel. You can configure which interfaces Yggdrasil beacons on with the `MulticastInterfaces` configuration directive. + +I wanted to know if we could blend the two so that Yggdrasil could automatically +discover other nearby devices and initiate peering connections with them using +AWDL. + +### Getting started + +Macs are a good target for developing and testing AWDL-aware applications as +AWDL is exposed to userspace through a network adapter called `awdl0`. It sits +there with a link-local IPv6 address, you can run `tcpdump` or Wireshark on it +to listen to AWDL traffic and you can even ping multicast group addresses on the +interface and get responses from other nearby devices, e.g. using `ping6 +ff02::1%awdl0`! However, Apple devices don't always keep AWDL alive and +listening all of the time. + +On macOS, the AWDL driver is only woken up when either AirDrop is being +actively used in Finder, or where a `NetService` has been created (usually +through Objective-C or Swift) which requests peer-to-peer networking. AWDL is +normally kept alive long enough to satisfy connectivity for these sessions and +then will be sent back to sleep after a period of idleness. + +On iOS, the story is somewhat similar to above, except that AWDL is often woken +up as soon as the device is unlocked if AirDrop is enabled. The `NetService` API +otherwise functions the same way. + +tvOS is the outlier in that it seems to wake up and listen to AWDL randomly, +even when the device is otherwise asleep, presumably because it is advertising +the ability to receive incoming AirPlay sessions to nearby devices. + +From a user perspective, the `awdl0` interface looks entirely unremarkable. It +behaves largely like any other ethernet interface, carrying regular IPv6 +traffic. In the background it's a bit more complicated, as the AWDL driver +performs traffic filtering for security reasons, namely, to stop someone sat +next to you in the airport from browsing your file shares. Namely, regular +listening sockets won't accept connections over AWDL unless a specific socket +option was configured on the socket before it started listening. + +Multicast traffic, however, does largely get passed through the filter +untouched. Bingo. + +### Waking up AWDL + +The `NetService` API is effectively a wrapper around multicast DNS-SD, which in +Apple's colourful language, is affectionately known as Bonjour. The API has the +added benefit of being able to tell the operating system to wake up the AWDL +driver pretty much on demand on behalf of "peer-to-peer" services. + +So all we would need to do to wake up AWDL is to call the `NetService` API, +publish a service that requests peer-to-peer functionality and let the operating +system do the hard work for us. Yggdrasil, being written in Go, didn't have any +concept of `NetService` but thankfully we were able to use Cgo to do this +instead. + +We wrote a Cgo function which calls the NetService API and advertises our new +fake service, `_yggdrasil._tcp`, which causes the operating system to wake up +the AWDL driver. Amazingly this worked. + +Yggdrasil doesn't actually use DNS-SD - we currently use a custom-formatted +multicast beacon on a different multicast group. It is planned to eventually +migrate to something more standard, like DNS-SD, for service discovery. However, +in this instance, registering a fake DNS-SD service was just enough to wake up +AWDL. + +### Peering automatically + +Once the driver is active, the regular Yggdrasil multicast beacons on the +`ff02::114` multicast group address seem to be passed through to the driver +normally and the Yggdrasil nodes running on each machine start to hear each +other's calls. + +The only thing that remained to be done was to configure the sockets with the +aforementioned socket option to allow them to communicate over the AWDL +interface. This socket option is called `SO_RECV_ANYIF` and is defined in +`sys/socket.h` on Darwin as `0x1104`. + +We configure the socket option on our TCP peering socket: +``` +err = unix.SetsockoptInt(int(fd), syscall.SOL_SOCKET, 0x1104, 1) +if err != nil { + ... +} +``` + +Now that the Yggdrasil nodes can hear each other's advertisements over the +`awdl0` interface, the regular automatic peering process kicks in and a TCP +session is opened between the two devices, creating a peering. The net result? +AWDL peerings! +``` +$ sudo yggdrasilctl getSwitchPeers + bytes_recvd bytes_sent coords endpoint ip port proto +1 304430 246994 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%en0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 1 tcp +2 1176278 878133 [3 5 5] fe80::xxxx:xxxx:xxxx:xxxx%en1 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 2 tcp +3 244278 313907 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%awdl0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 3 tcp +``` + +To further cement the experiment, we can actually disconnect the two devices +from each other, or connect to different Wi-Fi networks automatically, and the +peering over the `awdl0` interface still continues to function! + +An `iperf3` test over Yggdrasil using the new AWDL link looks fairly good - the +devices are sat next to each other: +``` +[ ID] Interval Transfer Bandwidth +[ 5] 0.00-1.00 sec 15.4 MBytes 129 Mbits/sec +[ 5] 1.00-2.00 sec 16.9 MBytes 141 Mbits/sec +[ 5] 2.00-3.00 sec 15.9 MBytes 133 Mbits/sec +[ 5] 3.00-4.00 sec 17.6 MBytes 147 Mbits/sec +[ 5] 4.00-5.00 sec 16.8 MBytes 141 Mbits/sec +[ 5] 5.00-6.00 sec 16.2 MBytes 136 Mbits/sec +[ 5] 6.00-7.00 sec 12.5 MBytes 105 Mbits/sec +[ 5] 7.00-8.00 sec 12.7 MBytes 106 Mbits/sec +[ 5] 8.00-9.00 sec 14.9 MBytes 125 Mbits/sec +[ 5] 9.00-10.00 sec 13.5 MBytes 113 Mbits/sec +``` + +### Observations and iOS + +As the `iperf3` test above shows, the link performance is actually quite good! +It routinely exceeds 100mbps, although this is between only two devices. I have +not been able to test this with Yggdrasil nodes running over AWDL in any +particular density due to only having a limited number of Macs to hand. + +One thing that I did notice though is that, while AWDL is active, my wireless +connection to my home Wi-Fi network does reduce in speed somewhat. This is to be +expected, given that the wireless chipset is hopping between channels rather +than spending all of it's time on a single channel. + +Sadly we weren't able to reproduce this test using iOS Testflight builds of +Yggdrasil. On iOS, we implement Yggdrasil as a VPN service which is subject to a +number of probably reasonable restrictions imposed by the OS, which presumably +exist to stop VPN extensions from spying on you. + +We were able to create a `NetService` from within the VPN extension and the +service beacons were advertised as expected, however, we weren't able to +initiate any other kind of connections over the `awdl0` interface. After a chat +with an engineer at Apple, it turns out that the `awdl0` interface isn't scoped +for use within a VPN extension, thus squashing our hopes and dreams of being +able to sprinkle this kind of magic onto our iOS port of Yggdrasil. We have a +feature request radar open with Apple in the hope that they may be able to +change this restriction in the future. + +But we were able to get this to work on macOS and that, itself, is quite +awesome. + +### Conclusion + +Yggdrasil doesn't enable AWDL by default because of the reduction in wireless +performance that AWDL being active can cause. Therefore, to enable AWDL peering, +you must add the `awdl0` interface specifically into the `MulticastInterfaces` +configuration option in `yggdrasil.conf`. However, we do have working support +for connecting Macs together and meshing automatically using AWDL, and you can +enable it very easily if you wish to experiment! + +We'd love to hear if you are peering Yggdrasil nodes using AWDL, or have +performed any more extensive tests of how it performs in real-world scenarios - +join us on our [Matrix channel](https://matrix.to/#/#yggdrasil:matrix.org)!