yggdrasil-network.github.io/_posts/2019-08-19-awdl.md

layout	title	date	author
post	Meshing using Apple Wireless Direct Link (AWDL)	2019-08-19 08:00:00 -0000	Neil Alexander

Wireless without borders

I was mostly prompted to write this post in response to a Hacker News thread recently, which announced the release of an open-source AirDrop implementation called OpenDrop, from the same team at Seemoo Lab who produced an open-source implementation of Apple Wireless Direct Link (AWDL) protocol called OWL. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless technologies. Even though everything covered in this post was done some time ago, I have never spent the time to document it.

With a few exceptions, most wireless networks in the world operate in "infrastructure mode" which is where a wireless access point serves one or more wireless clients. Think of your Wi-Fi at home, at work or in a coffee shop. However, as implied by the name, reliable and usable infrastructure Wi-Fi is often only available in certain physical locations with "good infrastructure". If you wanted to connect some devices together anywhere not served by an infrastructure Wi-Fi network, or in a location where you can't suddenly plug in a wireless access point, you may not have many options (Bluetooth aside).

AWDL is designed to avoid this problem by extending the 802.11 wireless standard to allow client devices to communicate directly with each other, without the help of the central wireless access point. You can walk out into a field with a couple of iPhones or Macs and they can use AWDL to discover each other and exchange data, peer-to-peer. Even better is that nearby devices that are connected to different infrastructure Wi-Fi networks can still communicate with each other using AWDL!

The science

Normally, when connected to a wireless access point, wireless clients remain locked to the specific radio channel that the AP is using. AWDL works by instructing the wireless adapter in the device to "hop" between channels so that it can not only remain connected to the wireless access point, but can also listen to other nearby devices.

Devices announce their presence and information about their services on a "social channel" for other devices to hear, effectively creating peer-to-peer service discovery. Once two devices have decided that they want to communicate directly, they agree to jump to another channel for real data exchange so that they don't interrupt existing Wi-Fi networks or, indeed, the social channel. These "hops" between wireless channels happen so quickly that there's very little disruption to what the user is doing with their Wi-Fi connection already (except for some minor wireless performance degradation - to be covered later).

A number of papers have been published by the OWLink team on the inner workings of the AWDL protocol, which can be found here. In particular, this paper from Mobicom 2018 contains a significant amount of detail about the AWDL protocol itself, channel hopping techniques and security considerations, amongst other things.

Mesh opportunities

Yggdrasil is designed to create a mesh network automatically out of interconnected nodes - the idea being that all nodes can route to all other nodes on the mesh network by routing through other nodes.

Today, many of these connections happen between nodes across the Internet, since the community is still relatively small and geographically dispersed. A node joining the Yggdrasil network needs to only peer with a single device that is already connected to the wider network in order to participate in the fully-routable mesh.

However, it's not the goal of Yggdrasil to remain something that we just toy with over the Internet. We want to build a protocol that can scale globally and work ad-hoc, even in places where infrastructure might not be particularly strong otherwise. We think that one of Yggdrasil's greatest strengths is that it is very close to zero-configuration, beyond giving it a very small number of configuration options, and it should scale well too in principle.

Yggdrasil can already discover potential peers on the same network segment by using multicast service discovery, which sounds a lot like what AWDL does on the social channel. You can configure which interfaces Yggdrasil beacons on with the MulticastInterfaces configuration directive.

I wanted to know if we could blend the two so that Yggdrasil could automatically discover other nearby devices and initiate peering connections with them using AWDL.

Getting started

Macs are a good target for developing and testing AWDL-aware applications as AWDL is exposed to userspace through a network adapter called awdl0. It sits there with a link-local IPv6 address, you can run tcpdump or Wireshark on it to listen to AWDL traffic and you can even ping multicast group addresses on the interface and get responses from other nearby devices, e.g. using ping6 ff02::1%awdl0! However, Apple devices don't always keep AWDL alive and listening all of the time.

On macOS, the AWDL driver is only woken up when either AirDrop is being actively used in Finder, or where a NetService has been created (usually through Objective-C or Swift) which requests peer-to-peer networking. AWDL is normally kept alive long enough to satisfy connectivity for these sessions and then will be sent back to sleep after a period of idleness.

On iOS, the story is somewhat similar to above, except that AWDL is often woken up as soon as the device is unlocked if AirDrop is enabled. The NetService API otherwise functions the same way.

tvOS is the outlier in that it seems to wake up and listen to AWDL randomly, even when the device is otherwise asleep, presumably because it is advertising the ability to receive incoming AirPlay sessions to nearby devices.

From a user perspective, the awdl0 interface looks entirely unremarkable. It behaves largely like any other ethernet interface, carrying regular IPv6 traffic. In the background it's a bit more complicated, as the AWDL driver performs traffic filtering for security reasons, namely, to stop someone sat next to you in the airport from browsing your file shares. Regular listening sockets won't accept connections over AWDL unless a specific socket option was configured on the socket before it started listening.

Multicast traffic, however, does largely get passed through the filter untouched. Bingo.

Waking up AWDL

The NetService API is effectively a wrapper around multicast DNS-SD, which in Apple's colourful language, is affectionately known as Bonjour. The API has the added benefit of being able to tell the operating system to wake up the AWDL driver pretty much on demand on behalf of "peer-to-peer" services.

So all we would need to do to wake up AWDL is to call the NetService API, publish a service that requests peer-to-peer functionality and let the operating system do the hard work for us. Yggdrasil, being written in Go, didn't have any concept of NetService but thankfully we were able to use Cgo to do this instead.

We wrote a Cgo function which calls the NetService API and advertises our new fake service, _yggdrasil._tcp, which causes the operating system to wake up the AWDL driver. Amazingly this worked.

Yggdrasil doesn't actually use DNS-SD - we currently use a custom-formatted multicast beacon on a different multicast group. It is planned to eventually migrate to something more standard, like DNS-SD, for service discovery. However, in this instance, registering a fake DNS-SD service was just enough to wake up AWDL.

Peering automatically

Once the driver is active, the regular Yggdrasil multicast beacons on the ff02::114 multicast group address seem to be passed through to the driver normally and the Yggdrasil nodes running on each machine start to hear each other's calls.

The only thing that remained to be done was to configure the sockets with the aforementioned socket option to allow them to communicate over the AWDL interface. This socket option is called SO_RECV_ANYIF and is defined in sys/socket.h on Darwin as 0x1104.

We configure the socket option on our TCP peering socket:

err = unix.SetsockoptInt(int(fd), syscall.SOL_SOCKET, 0x1104, 1)
if err != nil {
  ...
}

Now that the Yggdrasil nodes can hear each other's advertisements over the awdl0 interface, the regular automatic peering process kicks in and a TCP session is opened between the two devices, creating a peering. The net result? AWDL peerings!

$ sudo yggdrasilctl getSwitchPeers
   bytes_recvd   bytes_sent  coords       endpoint                         ip                                      port  proto  
1  244278        313907      [3 5 5 2 1]  fe80::xxxx:xxxx:xxxx:xxxx%awdl0  xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx  1     tcp

To further cement the experiment, we can actually disconnect the two devices from each other, or connect to different Wi-Fi networks automatically, and the peering over the awdl0 interface still continues to function!

An iperf3 test over Yggdrasil using the new AWDL link looks fairly good - the devices are sat next to each other:

[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   1.00-2.00   sec  16.9 MBytes   141 Mbits/sec                  
[  5]   2.00-3.00   sec  15.9 MBytes   133 Mbits/sec                  
[  5]   3.00-4.00   sec  17.6 MBytes   147 Mbits/sec                  
[  5]   4.00-5.00   sec  16.8 MBytes   141 Mbits/sec                  
[  5]   5.00-6.00   sec  16.2 MBytes   136 Mbits/sec                  
[  5]   6.00-7.00   sec  12.5 MBytes   105 Mbits/sec                  
[  5]   7.00-8.00   sec  12.7 MBytes   106 Mbits/sec                  
[  5]   8.00-9.00   sec  14.9 MBytes   125 Mbits/sec                  
[  5]   9.00-10.00  sec  13.5 MBytes   113 Mbits/sec                  

Observations and iOS

As the iperf3 test above shows, the link performance is actually quite good! It routinely exceeds 100mbps, although this is between only two devices. I have not been able to test this with Yggdrasil nodes running over AWDL in any particular density due to only having a limited number of Macs to hand.

One thing that I did notice though is that, while AWDL is active, my wireless connection to my home Wi-Fi network does reduce in speed somewhat. This is to be expected, given that the wireless chipset is hopping between channels rather than spending all of its time on a single channel.

Sadly we weren't able to reproduce this test using iOS Testflight builds of Yggdrasil. On iOS, we implement Yggdrasil as a VPN service which is subject to a number of probably reasonable restrictions imposed by the OS, which presumably exist to stop VPN extensions from spying on you.

We were able to create a NetService from within the VPN extension and the service beacons were advertised as expected, however, we weren't able to initiate any other kind of connections over the awdl0 interface. After a chat with an engineer at Apple, it turns out that the awdl0 interface isn't scoped for use within a VPN extension, thus squashing our hopes and dreams of being able to sprinkle this kind of magic onto our iOS port of Yggdrasil. We have a feature request radar open with Apple in the hope that they may be able to change this restriction in the future.

But we were able to get this to work on macOS and that, itself, is quite awesome.

Conclusion

Yggdrasil doesn't enable AWDL by default because of the reduction in wireless performance that AWDL being active can cause. Therefore, to enable AWDL peering, you must add the awdl0 interface specifically into the MulticastInterfaces configuration option in yggdrasil.conf. However, we do have working support for connecting Macs together and meshing automatically using AWDL, and you can enable it very easily if you wish to experiment!

We'd love to hear if you are peering Yggdrasil nodes using AWDL, or have performed any more extensive tests of how it performs in real-world scenarios - join us on our Matrix channel!