--- layout: post title: "Meshing using Apple Wireless Direct Link (AWDL)" date: 2019-08-19 08:00:00 -0000 author: Neil Alexander --- ### Wireless without borders I was mostly prompted to write this post in response to a [Hacker News thread](https://news.ycombinator.com/item?id=20735462) recently, which announced the release of an open-source implementation of the Apple Wireless Direct Link (AWDL) protocol. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless technologies. Even though everything covered in this post was done some time ago, I have never spent the time documenting it. With a few exceptions, most wireless networks in the world operate in "infrastructure mode" which is where a wireless access point serves one or more wireless clients. Think of your Wi-Fi at home, at work or in a coffee shop. However, as implied by the name, reliable and usable infrastructure Wi-Fi is often only available in certain physical locations with "good infrastructure". If you wanted to connect some devices together anywhere not served by an infrastructure Wi-Fi network, or in a location where you can't suddenly plug in a wireless access point, you may not have many options (Bluetooth aside). AWDL is designed to avoid this problem by extending the 802.11 wireless standard to allow client devices to communicate directly with each other, without the help of the central wireless access point. You can walk out into a field with a couple of iPhones or Macs and they can use AWDL to discover each other and exchange data, peer-to-peer. Even better is that nearby devices that are connected to different infrastructure Wi-Fi networks can still communicate with each other using AWDL! ### The science Normally, when connected to a wireless access point, wireless clients remain locked to the specific radio channel that the AP is using. AWDL works by instructing the wireless adapter in the device to "hop" between channels so that it can not only remain connected to the wireless access point, but can also listen to other nearby devices. Devices announce their presence and information about their services on a "social channel" for other devices to hear, effectively creating peer-to-peer service discovery. Once two devices have decided that they want to communicate directly, they agree to jump to another channel for real data exchange so that they don't interrupt existing Wi-Fi networks or, indeed, the social channel. These "hops" between wireless channels happen so quickly that there's very little disruption to what the user is doing with their Wi-Fi connection already (except for some minor wireless performance degradation - to be covered later). ### Mesh opportunities Yggdrasil is designed to create a mesh network automatically out of interconnected nodes - the idea being that all nodes can route to all other nodes on the mesh network by routing through other nodes. Today, many of these connections happen between nodes across the Internet, since the community is still relatively small and geographically dispersed. A node joining the Yggdrasil network needs to only peer with a single device that is already connected to the wider network in order to participate in the fully-routable mesh. However, it's not the goal of Yggdrasil to remain something that we just toy with over the Internet. We want to build a protocol that can scale globally and work ad-hoc, even in places where infrastructure might not be particularly strong otherwise. We think that one of Yggdrasil's greatest strengths is that it is very close to zero-configuration, beyond giving it a very small number of configuration options, and it should scale well too in principle. Yggdrasil can already discover potential peers on the same network segment by using multicast service discovery, which sounds a lot like what AWDL does on the social channel. You can configure which interfaces Yggdrasil beacons on with the `MulticastInterfaces` configuration directive. I wanted to know if we could blend the two so that Yggdrasil could automatically discover other nearby devices and initiate peering connections with them using AWDL. ### Getting started Macs are a good target for developing and testing AWDL-aware applications as AWDL is exposed to userspace through a network adapter called `awdl0`. It sits there with a link-local IPv6 address, you can run `tcpdump` or Wireshark on it to listen to AWDL traffic and you can even ping multicast group addresses on the interface and get responses from other nearby devices, e.g. using `ping6 ff02::1%awdl0`! However, Apple devices don't always keep AWDL alive and listening all of the time. On macOS, the AWDL driver is only woken up when either AirDrop is being actively used in Finder, or where a `NetService` has been created (usually through Objective-C or Swift) which requests peer-to-peer networking. AWDL is normally kept alive long enough to satisfy connectivity for these sessions and then will be sent back to sleep after a period of idleness. On iOS, the story is somewhat similar to above, except that AWDL is often woken up as soon as the device is unlocked if AirDrop is enabled. The `NetService` API otherwise functions the same way. tvOS is the outlier in that it seems to wake up and listen to AWDL randomly, even when the device is otherwise asleep, presumably because it is advertising the ability to receive incoming AirPlay sessions to nearby devices. From a user perspective, the `awdl0` interface looks entirely unremarkable. It behaves largely like any other ethernet interface, carrying regular IPv6 traffic. In the background it's a bit more complicated, as the AWDL driver performs traffic filtering for security reasons, namely, to stop someone sat next to you in the airport from browsing your file shares. Namely, regular listening sockets won't accept connections over AWDL unless a specific socket option was configured on the socket before it started listening. Multicast traffic, however, does largely get passed through the filter untouched. Bingo. ### Waking up AWDL The `NetService` API is effectively a wrapper around multicast DNS-SD, which in Apple's colourful language, is affectionately known as Bonjour. The API has the added benefit of being able to tell the operating system to wake up the AWDL driver pretty much on demand on behalf of "peer-to-peer" services. So all we would need to do to wake up AWDL is to call the `NetService` API, publish a service that requests peer-to-peer functionality and let the operating system do the hard work for us. Yggdrasil, being written in Go, didn't have any concept of `NetService` but thankfully we were able to use Cgo to do this instead. We wrote a Cgo function which calls the NetService API and advertises our new fake service, `_yggdrasil._tcp`, which causes the operating system to wake up the AWDL driver. Amazingly this worked. Yggdrasil doesn't actually use DNS-SD - we currently use a custom-formatted multicast beacon on a different multicast group. It is planned to eventually migrate to something more standard, like DNS-SD, for service discovery. However, in this instance, registering a fake DNS-SD service was just enough to wake up AWDL. ### Peering automatically Once the driver is active, the regular Yggdrasil multicast beacons on the `ff02::114` multicast group address seem to be passed through to the driver normally and the Yggdrasil nodes running on each machine start to hear each other's calls. The only thing that remained to be done was to configure the sockets with the aforementioned socket option to allow them to communicate over the AWDL interface. This socket option is called `SO_RECV_ANYIF` and is defined in `sys/socket.h` on Darwin as `0x1104`. We configure the socket option on our TCP peering socket: ``` err = unix.SetsockoptInt(int(fd), syscall.SOL_SOCKET, 0x1104, 1) if err != nil { ... } ``` Now that the Yggdrasil nodes can hear each other's advertisements over the `awdl0` interface, the regular automatic peering process kicks in and a TCP session is opened between the two devices, creating a peering. The net result? AWDL peerings! ``` $ sudo yggdrasilctl getSwitchPeers bytes_recvd bytes_sent coords endpoint ip port proto 1 244278 313907 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%awdl0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 1 tcp ``` To further cement the experiment, we can actually disconnect the two devices from each other, or connect to different Wi-Fi networks automatically, and the peering over the `awdl0` interface still continues to function! An `iperf3` test over Yggdrasil using the new AWDL link looks fairly good - the devices are sat next to each other: ``` [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 15.4 MBytes 129 Mbits/sec [ 5] 1.00-2.00 sec 16.9 MBytes 141 Mbits/sec [ 5] 2.00-3.00 sec 15.9 MBytes 133 Mbits/sec [ 5] 3.00-4.00 sec 17.6 MBytes 147 Mbits/sec [ 5] 4.00-5.00 sec 16.8 MBytes 141 Mbits/sec [ 5] 5.00-6.00 sec 16.2 MBytes 136 Mbits/sec [ 5] 6.00-7.00 sec 12.5 MBytes 105 Mbits/sec [ 5] 7.00-8.00 sec 12.7 MBytes 106 Mbits/sec [ 5] 8.00-9.00 sec 14.9 MBytes 125 Mbits/sec [ 5] 9.00-10.00 sec 13.5 MBytes 113 Mbits/sec ``` ### Observations and iOS As the `iperf3` test above shows, the link performance is actually quite good! It routinely exceeds 100mbps, although this is between only two devices. I have not been able to test this with Yggdrasil nodes running over AWDL in any particular density due to only having a limited number of Macs to hand. One thing that I did notice though is that, while AWDL is active, my wireless connection to my home Wi-Fi network does reduce in speed somewhat. This is to be expected, given that the wireless chipset is hopping between channels rather than spending all of it's time on a single channel. Sadly we weren't able to reproduce this test using iOS Testflight builds of Yggdrasil. On iOS, we implement Yggdrasil as a VPN service which is subject to a number of probably reasonable restrictions imposed by the OS, which presumably exist to stop VPN extensions from spying on you. We were able to create a `NetService` from within the VPN extension and the service beacons were advertised as expected, however, we weren't able to initiate any other kind of connections over the `awdl0` interface. After a chat with an engineer at Apple, it turns out that the `awdl0` interface isn't scoped for use within a VPN extension, thus squashing our hopes and dreams of being able to sprinkle this kind of magic onto our iOS port of Yggdrasil. We have a feature request radar open with Apple in the hope that they may be able to change this restriction in the future. But we were able to get this to work on macOS and that, itself, is quite awesome. ### Conclusion Yggdrasil doesn't enable AWDL by default because of the reduction in wireless performance that AWDL being active can cause. Therefore, to enable AWDL peering, you must add the `awdl0` interface specifically into the `MulticastInterfaces` configuration option in `yggdrasil.conf`. However, we do have working support for connecting Macs together and meshing automatically using AWDL, and you can enable it very easily if you wish to experiment! We'd love to hear if you are peering Yggdrasil nodes using AWDL, or have performed any more extensive tests of how it performs in real-world scenarios - join us on our [Matrix channel](https://matrix.to/#/#yggdrasil:matrix.org)!