yggdrasil-network.github.io/_posts/2019-08-19-awdl.md

---
layout: post
title:  "Meshing using Apple Wireless Direct Link (AWDL)"
date:   2019-08-19 08:00:00 -0000
author: Neil Alexander
---

### Wireless without borders

I was mostly prompted to write this post in response to a [Hacker News
thread](https://news.ycombinator.com/item?id=20735462) recently, which announced
the release of an open-source AirDrop implementation called
[OpenDrop](https://github.com/seemoo-lab/opendrop), from the same team at Seemoo
Lab who produced an open-source implementation of Apple Wireless Direct Link
(AWDL) protocol called [OWL](https://github.com/seemoo-lab/owl). AWDL is the
secret sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless
technologies. Even though everything covered in this post was done some time
ago, I have never spent the time to document it.

With a few exceptions, most wireless networks in the world operate in
"infrastructure mode" which is where a wireless access point serves one or more
wireless clients. Think of your Wi-Fi at home, at work or in a coffee shop.
However, as implied by the name, reliable and usable infrastructure Wi-Fi is
often only available in certain physical locations with "good infrastructure".
If you wanted to connect some devices together anywhere not served by an
infrastructure Wi-Fi network, or in a location where you can't suddenly plug in
a wireless access point, you may not have many options (Bluetooth aside).

AWDL is designed to avoid this problem by extending the 802.11 wireless standard
to allow client devices to communicate directly with each other, without the
help of the central wireless access point. You can walk out into a field with a
couple of iPhones or Macs and they can use AWDL to discover each other and
exchange data, peer-to-peer. Even better is that nearby devices that are
connected to different infrastructure Wi-Fi networks can still communicate with
each other using AWDL!

### The science

Normally, when connected to a wireless access point, wireless clients remain
locked to the specific radio channel that the AP is using. AWDL works by
instructing the wireless adapter in the device to "hop" between channels so that
it can not only remain connected to the wireless access point, but can also
listen to other nearby devices.

Devices announce their presence and information about their services on a
"social channel" for other devices to hear, effectively creating peer-to-peer
service discovery. Once two devices have decided that they want to communicate
directly, they agree to jump to another channel for real data exchange so that
they don't interrupt existing Wi-Fi networks or, indeed, the social channel.
These "hops" between wireless channels happen so quickly that there's very
little disruption to what the user is doing with their Wi-Fi connection already
(except for some minor wireless performance degradation - to be covered later).

A number of papers have been published by the OWLink team on the inner workings
of the AWDL protocol, which can be [found
here](https://owlink.org/publications/). In particular, [this
paper](https://arxiv.org/pdf/1808.03156.pdf) from Mobicom 2018 contains a
significant amount of detail about the AWDL protocol itself, channel hopping
techniques and security considerations, amongst other things.

### Mesh opportunities

Yggdrasil is designed to create a mesh network automatically out of
interconnected nodes - the idea being that all nodes can route to all other
nodes on the mesh network by routing through other nodes.

Today, many of these connections happen between nodes across the Internet, since
the community is still relatively small and geographically dispersed. A node
joining the Yggdrasil network needs to only peer with a single device that is
already connected to the wider network in order to participate in the
fully-routable mesh.

However, it's not the goal of Yggdrasil to remain something that we just toy
with over the Internet. We want to build a protocol that can scale globally and
work ad-hoc, even in places where infrastructure might not be particularly
strong otherwise. We think that one of Yggdrasil's greatest strengths is that it
is very close to zero-configuration, beyond giving it a very small number of
configuration options, and it should scale well too in principle.

Yggdrasil can already discover potential peers on the same network segment by
using multicast service discovery, which sounds a lot like what AWDL does on the
social channel. You can configure which interfaces Yggdrasil beacons on with the
`MulticastInterfaces` configuration directive.

I wanted to know if we could blend the two so that Yggdrasil could automatically
discover other nearby devices and initiate peering connections with them using
AWDL.

### Getting started

Macs are a good target for developing and testing AWDL-aware applications as
AWDL is exposed to userspace through a network adapter called `awdl0`. It sits
there with a link-local IPv6 address, you can run `tcpdump` or Wireshark on it
to listen to AWDL traffic and you can even ping multicast group addresses on the
interface and get responses from other nearby devices, e.g. using `ping6
ff02::1%awdl0`! However, Apple devices don't always keep AWDL alive and
listening all of the time.

On macOS, the AWDL driver is only woken up when either AirDrop is being
actively used in Finder, or where a `NetService` has been created (usually
through Objective-C or Swift) which requests peer-to-peer networking. AWDL is
normally kept alive long enough to satisfy connectivity for these sessions and
then will be sent back to sleep after a period of idleness.

On iOS, the story is somewhat similar to above, except that AWDL is often woken
up as soon as the device is unlocked if AirDrop is enabled. The `NetService` API
otherwise functions the same way.

tvOS is the outlier in that it seems to wake up and listen to AWDL randomly,
even when the device is otherwise asleep, presumably because it is advertising
the ability to receive incoming AirPlay sessions to nearby devices.

From a user perspective, the `awdl0` interface looks entirely unremarkable. It
behaves largely like any other ethernet interface, carrying regular IPv6
traffic. In the background it's a bit more complicated, as the AWDL driver
performs traffic filtering for security reasons, namely, to stop someone sat
next to you in the airport from browsing your file shares. Regular listening
sockets won't accept connections over AWDL unless a specific socket option was
configured on the socket before it started listening.

Multicast traffic, however, does largely get passed through the filter
untouched. Bingo.

### Waking up AWDL

The `NetService` API is effectively a wrapper around multicast DNS-SD, which in
Apple's colourful language, is affectionately known as Bonjour. The API has the
added benefit of being able to tell the operating system to wake up the AWDL
driver pretty much on demand on behalf of "peer-to-peer" services.

So all we would need to do to wake up AWDL is to call the `NetService` API,
publish a service that requests peer-to-peer functionality and let the operating
system do the hard work for us. Yggdrasil, being written in Go, didn't have any
concept of `NetService` but thankfully we were able to use Cgo to do this
instead.

We wrote a Cgo function which calls the NetService API and advertises our new
fake service, `_yggdrasil._tcp`, which causes the operating system to wake up
the AWDL driver. Amazingly this worked.

Yggdrasil doesn't actually use DNS-SD - we currently use a custom-formatted
multicast beacon on a different multicast group. It is planned to eventually
migrate to something more standard, like DNS-SD, for service discovery. However,
in this instance, registering a fake DNS-SD service was just enough to wake up
AWDL.

### Peering automatically

Once the driver is active, the regular Yggdrasil multicast beacons on the
`ff02::114` multicast group address seem to be passed through to the driver
normally and the Yggdrasil nodes running on each machine start to hear each
other's calls.

The only thing that remained to be done was to configure the sockets with the
aforementioned socket option to allow them to communicate over the AWDL
interface. This socket option is called `SO_RECV_ANYIF` and is defined in
`sys/socket.h` on Darwin as `0x1104`.

We configure the socket option on our TCP peering socket:
```
err = unix.SetsockoptInt(int(fd), syscall.SOL_SOCKET, 0x1104, 1)
if err != nil {
  ...
}
```

Now that the Yggdrasil nodes can hear each other's advertisements over the
`awdl0` interface, the regular automatic peering process kicks in and a TCP
session is opened between the two devices, creating a peering. The net result?
AWDL peerings!
```
$ sudo yggdrasilctl getSwitchPeers
   bytes_recvd   bytes_sent  coords       endpoint                         ip                                      port  proto  
1  244278        313907      [3 5 5 2 1]  fe80::xxxx:xxxx:xxxx:xxxx%awdl0  xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx  1     tcp
```

To further cement the experiment, we can actually disconnect the two devices
from each other, or connect to different Wi-Fi networks automatically, and the
peering over the `awdl0` interface still continues to function!

An `iperf3` test over Yggdrasil using the new AWDL link looks fairly good - the
devices are sat next to each other:
```
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  15.4 MBytes   129 Mbits/sec                  
[  5]   1.00-2.00   sec  16.9 MBytes   141 Mbits/sec                  
[  5]   2.00-3.00   sec  15.9 MBytes   133 Mbits/sec                  
[  5]   3.00-4.00   sec  17.6 MBytes   147 Mbits/sec                  
[  5]   4.00-5.00   sec  16.8 MBytes   141 Mbits/sec                  
[  5]   5.00-6.00   sec  16.2 MBytes   136 Mbits/sec                  
[  5]   6.00-7.00   sec  12.5 MBytes   105 Mbits/sec                  
[  5]   7.00-8.00   sec  12.7 MBytes   106 Mbits/sec                  
[  5]   8.00-9.00   sec  14.9 MBytes   125 Mbits/sec                  
[  5]   9.00-10.00  sec  13.5 MBytes   113 Mbits/sec                  
```

### Observations and iOS

As the `iperf3` test above shows, the link performance is actually quite good!
It routinely exceeds 100mbps, although this is between only two devices. I have
not been able to test this with Yggdrasil nodes running over AWDL in any
particular density due to only having a limited number of Macs to hand.

One thing that I did notice though is that, while AWDL is active, my wireless
connection to my home Wi-Fi network does reduce in speed somewhat. This is to be
expected, given that the wireless chipset is hopping between channels rather
than spending all of its time on a single channel.

Sadly we weren't able to reproduce this test using iOS Testflight builds of
Yggdrasil. On iOS, we implement Yggdrasil as a VPN service which is subject to a
number of probably reasonable restrictions imposed by the OS, which presumably
exist to stop VPN extensions from spying on you.

We were able to create a `NetService` from within the VPN extension and the
service beacons were advertised as expected, however, we weren't able to
initiate any other kind of connections over the `awdl0` interface. After a chat
with an engineer at Apple, it turns out that the `awdl0` interface isn't scoped
for use within a VPN extension, thus squashing our hopes and dreams of being
able to sprinkle this kind of magic onto our iOS port of Yggdrasil. We have a
feature request radar open with Apple in the hope that they may be able to
change this restriction in the future.

But we were able to get this to work on macOS and that, itself, is quite
awesome.

### Conclusion

Yggdrasil doesn't enable AWDL by default because of the reduction in wireless
performance that AWDL being active can cause. Therefore, to enable AWDL peering,
you must add the `awdl0` interface specifically into the `MulticastInterfaces`
configuration option in `yggdrasil.conf`. However, we do have working support
for connecting Macs together and meshing automatically using AWDL, and you can
enable it very easily if you wish to experiment!

We'd love to hear if you are peering Yggdrasil nodes using AWDL, or have
performed any more extensive tests of how it performs in real-world scenarios -
join us on our [Matrix channel](https://matrix.to/#/#yggdrasil:matrix.org)!