From 62e61ce65e4980fb8d83190f25b6fe262c5c5bb4 Mon Sep 17 00:00:00 2001 From: ghbjklhv <50426427+ghbjklhv@users.noreply.github.com> Date: Wed, 19 Jun 2019 03:02:54 +0000 Subject: [PATCH 01/95] Update faq.md --- faq.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/faq.md b/faq.md index 13a69d2..f637c00 100644 --- a/faq.md +++ b/faq.md @@ -83,6 +83,16 @@ yggdrasilctl -endpoint=127.0.0.1:12345 You can set the `IfName` configuration setting to `"none"`. This will load Yggdrasil, but will not create a TUN/TAP adapter, meaning that your host will not be exposed to the Yggdrasil network. This does of course mean that you won't be able to send any traffic from that node to Yggdrasil either! +### Does Yggdrasil work with my VPN? + +VPN traffic can be sent while using Yggdrasil. + +### Does Yggdrasil work with networks like i2p and TOR? + +i2p and TOR work over Yggdrasil. +See: [/public-peers/tree/master/other](https://github.com/yggdrasil-network/public-peers/tree/master/other) + + ### I want to allow outgoing connections from my machine but prevent unwanted incoming connections. Generally this requires you to use a firewall. The steps for this will vary from platform to platform. From 436f9af78c3d38240a5b75d16f5a82fe753afe66 Mon Sep 17 00:00:00 2001 From: ghbjklhv <50426427+ghbjklhv@users.noreply.github.com> Date: Wed, 19 Jun 2019 03:05:10 +0000 Subject: [PATCH 02/95] Update faq.md --- faq.md | 1 + 1 file changed, 1 insertion(+) diff --git a/faq.md b/faq.md index f637c00..e6c11f1 100644 --- a/faq.md +++ b/faq.md @@ -90,6 +90,7 @@ VPN traffic can be sent while using Yggdrasil. ### Does Yggdrasil work with networks like i2p and TOR? i2p and TOR work over Yggdrasil. + See: [/public-peers/tree/master/other](https://github.com/yggdrasil-network/public-peers/tree/master/other) From 22fdc661d636ea7b120a760b036ff194c88d4197 Mon Sep 17 00:00:00 2001 From: ghbjklhv <50426427+ghbjklhv@users.noreply.github.com> Date: Wed, 19 Jun 2019 03:06:39 +0000 Subject: [PATCH 03/95] Update faq.md --- faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/faq.md b/faq.md index e6c11f1..7a53f5d 100644 --- a/faq.md +++ b/faq.md @@ -89,7 +89,7 @@ VPN traffic can be sent while using Yggdrasil. ### Does Yggdrasil work with networks like i2p and TOR? -i2p and TOR work over Yggdrasil. +i2p and TOR work with Yggdrasil. See: [/public-peers/tree/master/other](https://github.com/yggdrasil-network/public-peers/tree/master/other) From c80f5c93586da5bc130494d80e739d0ece7a949b Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 19 Jun 2019 22:45:58 +0100 Subject: [PATCH 04/95] Update installation-linux-deb.md --- installation-linux-deb.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/installation-linux-deb.md b/installation-linux-deb.md index c4f61d9..0facb48 100644 --- a/installation-linux-deb.md +++ b/installation-linux-deb.md @@ -22,13 +22,13 @@ sudo apt-get install dirmngr Then import the repository key to your `gpg` keyring and export it to your `apt` keyring: ``` -gpg --fetch-keys https://neilalexander.s3.eu-west-2.amazonaws.com/deb/key.txt +gpg --fetch-keys https://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/deb/key.txt gpg --export 569130E8CA20FBC4CB3FDE555898470A764B32C9 | sudo apt-key add - ``` Add the repository into your `apt` sources: ``` -echo 'deb http://neilalexander.s3.eu-west-2.amazonaws.com/deb/ debian yggdrasil' | sudo tee /etc/apt/sources.list.d/yggdrasil.list +echo 'deb http://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/deb/ debian yggdrasil' | sudo tee /etc/apt/sources.list.d/yggdrasil.list sudo apt-get update ``` From c527698996649446ee7541a3f3e606380636dee8 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 19 Jun 2019 22:46:15 +0100 Subject: [PATCH 05/95] Update installation-linux-edgeos.md --- installation-linux-edgeos.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/installation-linux-edgeos.md b/installation-linux-edgeos.md index 0087e3a..bbd4eb3 100644 --- a/installation-linux-edgeos.md +++ b/installation-linux-edgeos.md @@ -16,7 +16,7 @@ configured. Start by adding the GPG key: ``` -curl -o- https://neilalexander.s3.eu-west-2.amazonaws.com/deb/key.txt | sudo apt-key add - +curl -o- https://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/deb/key.txt | sudo apt-key add - ``` Then add the repository: @@ -24,7 +24,7 @@ Then add the repository: configure set system package repository yggdrasil distribution edgerouter set system package repository yggdrasil components yggdrasil -set system package repository yggdrasil url http://neilalexander.s3.eu-west-2.amazonaws.com/deb/ +set system package repository yggdrasil url http://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/deb/ commit save ``` From 4e9401d1e836e06ceb3aeedc19fea086ca99c1f6 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 19 Jun 2019 22:46:38 +0100 Subject: [PATCH 06/95] Update installation-linux-rpm.md --- installation-linux-rpm.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/installation-linux-rpm.md b/installation-linux-rpm.md index 725997c..12a55ba 100644 --- a/installation-linux-rpm.md +++ b/installation-linux-rpm.md @@ -15,7 +15,7 @@ will also work on any other RPM-based distribution. To start with, import the repository key to your `gpg` keyring and export it to your `rpm` keyring: ``` -gpg --fetch-keys https://neilalexander.s3.eu-west-2.amazonaws.com/deb/key.txt +gpg --fetch-keys https://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/deb/key.txt gpg --armor --no-comment --export-options export-minimal --export 569130E8CA20FBC4CB3FDE555898470A764B32C9 | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-yggdrasil ``` @@ -24,7 +24,7 @@ Add the repository: sudo cat > /etc/yum.repos.d/yggdrasil.repo << EOF [yggdrasil] name = Yggdrasil -baseurl = https://neilalexander.s3.eu-west-2.amazonaws.com/rpm/ +baseurl = https://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/rpm/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-yggdrasil EOF From 42cac3502e168e87dedc4397592e76eb3fd1aa77 Mon Sep 17 00:00:00 2001 From: ne-vlezay80 Date: Tue, 25 Jun 2019 20:26:22 +0300 Subject: [PATCH 07/95] Add new service http://[300:aa35:f9c1:dcce::2]/ - IPv6 address information from yggdrasil network. --- services.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/services.md b/services.md index 2e90aba..a9578b0 100644 --- a/services.md +++ b/services.md @@ -34,6 +34,7 @@ The following services are available on the Yggdrasil network, courtesy of our c ---- + ## Shell accounts ### BAN.AI Public Access Multics Service @@ -105,6 +106,9 @@ The following services are available on the Yggdrasil network, courtesy of our c - [Arceliar's Interactive Network Map](http://[21f:dd73:7cdb:773b:a924:7ec0:800b:221e]) - [y.yakamo.org Static World Map](http://[301:4541:2f84:1188:216:3eff:feb6:65a3]:3000/static/map.png) +### IPv6 address information +- http://[300:aa35:f9c1:dcce::2]/ + ---- ## BitTorrent From d45e8242096713bf0947766eb2dccae01a20f5a9 Mon Sep 17 00:00:00 2001 From: ne-vlezay80 Date: Fri, 28 Jun 2019 23:23:12 +0300 Subject: [PATCH 08/95] Add new game service --- services.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/services.md b/services.md index 2e90aba..fe095f0 100644 --- a/services.md +++ b/services.md @@ -126,6 +126,9 @@ The following services are available on the Yggdrasil network, courtesy of our c - `202:baed:9808:83c8:738:4041:bd8b:8c1d`, hosted by piele - DNS name: `mc.y.creativeserver.net` - Survival mode map, ask piele to be whitelisted + +### Minetest + - `300:aa35:f9c1:dcce::3:30000`, admin Admin ---- From 377d91032947d7ed78a5123ff5ec14bb61e5f484 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 15 Jul 2019 14:30:20 +0100 Subject: [PATCH 09/95] Update faq.md --- faq.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/faq.md b/faq.md index 7a53f5d..aee6ff1 100644 --- a/faq.md +++ b/faq.md @@ -83,16 +83,13 @@ yggdrasilctl -endpoint=127.0.0.1:12345 You can set the `IfName` configuration setting to `"none"`. This will load Yggdrasil, but will not create a TUN/TAP adapter, meaning that your host will not be exposed to the Yggdrasil network. This does of course mean that you won't be able to send any traffic from that node to Yggdrasil either! -### Does Yggdrasil work with my VPN? +### Does Yggdrasil work alongside an existing VPN? -VPN traffic can be sent while using Yggdrasil. +Yes, Yggdrasil should not interfere with existing VPNs. VPN traffic can be sent while using Yggdrasil. ### Does Yggdrasil work with networks like i2p and TOR? -i2p and TOR work with Yggdrasil. - -See: [/public-peers/tree/master/other](https://github.com/yggdrasil-network/public-peers/tree/master/other) - +Yggdrasil can peer over Tor or I2P. See [/public-peers/tree/master/other](https://github.com/yggdrasil-network/public-peers/tree/master/other) for public peers. ### I want to allow outgoing connections from my machine but prevent unwanted incoming connections. From 0bad18733786a6e6e41340b5526c2102dbfb9480 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 15 Jul 2019 14:33:14 +0100 Subject: [PATCH 10/95] Update faq.md --- faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/faq.md b/faq.md index aee6ff1..e0c65e8 100644 --- a/faq.md +++ b/faq.md @@ -87,7 +87,7 @@ You can set the `IfName` configuration setting to `"none"`. This will load Yggdr Yes, Yggdrasil should not interfere with existing VPNs. VPN traffic can be sent while using Yggdrasil. -### Does Yggdrasil work with networks like i2p and TOR? +### Does Yggdrasil work with networks like Tor or I2P? Yggdrasil can peer over Tor or I2P. See [/public-peers/tree/master/other](https://github.com/yggdrasil-network/public-peers/tree/master/other) for public peers. From f323da9a0eabe027e538e38e06108910e655783e Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sat, 20 Jul 2019 11:28:57 +0100 Subject: [PATCH 11/95] Windows installation page --- installation-windows.md | 59 +++++++++++++++++++++++++++++++++++++++++ installation.md | 4 +++ platform-windows.md | 55 -------------------------------------- 3 files changed, 63 insertions(+), 55 deletions(-) create mode 100644 installation-windows.md diff --git a/installation-windows.md b/installation-windows.md new file mode 100644 index 0000000..f6f924c --- /dev/null +++ b/installation-windows.md @@ -0,0 +1,59 @@ +--- +tags: dontlink +sitemap: true +--- + +## TAP Driver + +Yggdrasil depends on the OpenVPN TAP driver to work on Windows. There are two flavours of this driver: + +- [NDIS 5](https://swupdate.openvpn.org/community/releases/tap-windows-9.9.2_3.exe) (`tap-windows-9.9.2_3`) - recommended +- [NDIS 6](https://swupdate.openvpn.org/community/releases/tap-windows-9.21.2.exe) (`tap-windows-9.21.2`) - functional + +Yggdrasil works with either driver, although the performance is remarkably better with the older NDIS 5 driver, therefore it is recommended to use that instead of the NDIS 6 driver. + +Please note that if you already have OpenVPN for Windows, you likely have one of the two drivers installed already and should not install it again. + +## Installation + +Start by [downloading the latest Yggdrasil binary](builds.md) and use Windows Explorer to copy it into a sensible location and rename it to `yggdrasil.exe`. + +The below examples assume that you placed `yggdrasil.exe` into `C:\Program Files\Yggdrasil`. + +## Generate configuration + +Before starting Yggdrasil, you should generate configuration. Open a Command Prompt as Administrator: +``` +"C:\Program Files\Yggdrasil\yggdrasil.exe" -genconf > "C:\Program Files\Yggdrasil\yggdrasil.conf" +``` + +## Run Yggdrasil + +### Run once + +Open a Command Prompt as Administrator and start Yggdrasil using your generated configuration: +``` +"C:\Program Files\Yggdrasil\yggdrasil.exe" -useconffile "C:\Program Files\Yggdrasil\yggdrasil.conf" +``` +Alternatively, start Yggdrasil in auto-configuration mode: +``` +"C:\Program Files\Yggdrasil\yggdrasil.exe" -autoconf +``` + +### Run as a background service + +Running as a background system service means that Yggdrasil will automatically start up in the background when Windows boots. + +Assuming that Yggdrasil is installed into `C:\Program Files\Yggdrasil` and your configuration *already exists* in `C:\Program Files\Yggdrasil\yggdrasil.conf`, as above, then you can install Yggdrasil as a Windows service. Open a Command Prompt as Administrator: +``` +sc create yggdrasil binpath= "\"C:\Program Files\Yggdrasil\yggdrasil.exe\" -useconffile \"C:\Program Files\Yggdrasil\yggdrasil.conf\"" +sc config yggdrasil displayname= "Yggdrasil Service" +sc config yggdrasil start= "auto" +sc start yggdrasil +``` +Alternatively, if you want the service to autoconfigure instead of using an `yggdrasil.conf`, replace the `sc create` line from above with: +``` +sc create yggdrasil binpath= "\"C:\Program Files\Yggdrasil\yggdrasil.exe\" -autoconf" +``` + +The Yggdrasil service can then be stopped and started using `services.msc`, or in more recent versions of Windows, the Task Manager. diff --git a/installation.md b/installation.md index f6e5c94..cced1eb 100644 --- a/installation.md +++ b/installation.md @@ -21,3 +21,7 @@ Installation guides are available for the following platforms/distributions. ## iOS - [App installation](installation-ios-app.md) + +## Windows + +- [Manual install as a Windows service](installation-windows.md) diff --git a/platform-windows.md b/platform-windows.md index 1584c7a..626fc2e 100644 --- a/platform-windows.md +++ b/platform-windows.md @@ -14,61 +14,6 @@ Yggdrasil has been tested and is working on the following versions of Windows: It may work on other versions of Windows too, although these are untested. -## TAP Driver - -Yggdrasil depends on the OpenVPN TAP driver to work on Windows. There are two flavours of this driver: - -- [NDIS 5](https://swupdate.openvpn.org/community/releases/tap-windows-9.9.2_3.exe) (`tap-windows-9.9.2_3`) - recommended -- [NDIS 6](https://swupdate.openvpn.org/community/releases/tap-windows-9.21.2.exe) (`tap-windows-9.21.2`) - functional - -Yggdrasil works with either driver, although the performance is remarkably better with the older NDIS 5 driver, therefore it is recommended to use that instead of the NDIS 6 driver. - -Please note that if you already have OpenVPN for Windows, you likely have one of the two drivers installed already and should not install it again. - ## Notes - TUN mode is not supported on Windows, and attempts to configure TUN mode will default to TAP mode automatically. - -## Installation - -Start by [downloading the latest Yggdrasil binary](builds.md) and use Windows Explorer to copy it into a sensible location and rename it to `yggdrasil.exe`. - -The below examples assume that you placed `yggdrasil.exe` into `C:\Program Files\Yggdrasil`. - -## Generate configuration - -Before starting Yggdrasil, you should generate configuration. Open a Command Prompt as Administrator: -``` -"C:\Program Files\Yggdrasil\yggdrasil.exe" -genconf > "C:\Program Files\Yggdrasil\yggdrasil.conf" -``` - -## Run Yggdrasil - -### Run once - -Open a Command Prompt as Administrator and start Yggdrasil using your generated configuration: -``` -"C:\Program Files\Yggdrasil\yggdrasil.exe" -useconffile "C:\Program Files\Yggdrasil\yggdrasil.conf" -``` -Alternatively, start Yggdrasil in auto-configuration mode: -``` -"C:\Program Files\Yggdrasil\yggdrasil.exe" -autoconf -``` - -### Run as a background service - -Running as a background system service means that Yggdrasil will automatically start up in the background when Windows boots. - -Assuming that Yggdrasil is installed into `C:\Program Files\Yggdrasil` and your configuration *already exists* in `C:\Program Files\Yggdrasil\yggdrasil.conf`, as above, then you can install Yggdrasil as a Windows service. Open a Command Prompt as Administrator: -``` -sc create yggdrasil binpath= "\"C:\Program Files\Yggdrasil\yggdrasil.exe\" -useconffile \"C:\Program Files\Yggdrasil\yggdrasil.conf\"" -sc config yggdrasil displayname= "Yggdrasil Service" -sc config yggdrasil start= "auto" -sc start yggdrasil -``` -Alternatively, if you want the service to autoconfigure instead of using an `yggdrasil.conf`, replace the `sc create` line from above with: -``` -sc create yggdrasil binpath= "\"C:\Program Files\Yggdrasil\yggdrasil.exe\" -autoconf" -``` - -The Yggdrasil service can then be stopped and started using `services.msc`, or in more recent versions of Windows, the Task Manager. From 9ea808bff9f71674d4face9b89be90a162b934ac Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sat, 20 Jul 2019 11:31:48 +0100 Subject: [PATCH 12/95] Add header to Windows installation page --- installation-windows.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/installation-windows.md b/installation-windows.md index f6f924c..0aa0a57 100644 --- a/installation-windows.md +++ b/installation-windows.md @@ -3,6 +3,10 @@ tags: dontlink sitemap: true --- +# Installing manually on Windows + +Yggdrasil is supported on Windows. You can either [download the latest binary from CircleCI](builds.md). + ## TAP Driver Yggdrasil depends on the OpenVPN TAP driver to work on Windows. There are two flavours of this driver: From 95543106b56603a8f40446e0509e5a972f51aa8c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Per=20=28=E1=B5=94=E1=B4=A5=E1=B5=94=29?= Date: Sat, 20 Jul 2019 21:11:22 +0200 Subject: [PATCH 13/95] Public iPerf3 service on Ygg. --- services.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/services.md b/services.md index 2f80428..c553624 100644 --- a/services.md +++ b/services.md @@ -181,3 +181,9 @@ The following services are available on the Yggdrasil network, courtesy of our c - DC-hub by *whoami* `adc://[202:9877:2815:cd91:336:2a16:bfd9:7258]:1511` - Works with clients who support IPv6 and ADC. Ncdc or Airdcpp for example (eiskaltdc++ not support IPv6) + +---- + +## iPerf3 + +- `iperf3 -6 -c y.thingylabs.io` on [ygg.thingylabs.io](https://ygg.thingylabs.io/) From f9c4999dcfe50cf00badcd9802b6ab41bd82e57b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Per=20=28=E1=B5=94=E1=B4=A5=E1=B5=94=29?= Date: Sat, 20 Jul 2019 21:19:58 +0200 Subject: [PATCH 14/95] Location and link speed. --- services.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/services.md b/services.md index c553624..7df9e14 100644 --- a/services.md +++ b/services.md @@ -187,3 +187,5 @@ The following services are available on the Yggdrasil network, courtesy of our c ## iPerf3 - `iperf3 -6 -c y.thingylabs.io` on [ygg.thingylabs.io](https://ygg.thingylabs.io/) + - Link: 10 GBit + - Location: Nuremberg, Germany From 1a336497d0222ff90b9aa7f3a9700e47b6b6a673 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Per=20=28=E1=B5=94=E1=B4=A5=E1=B5=94=29?= Date: Sat, 20 Jul 2019 21:54:19 +0200 Subject: [PATCH 15/95] Link the tool it's about. --- services.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/services.md b/services.md index 7df9e14..422f6f9 100644 --- a/services.md +++ b/services.md @@ -184,7 +184,7 @@ The following services are available on the Yggdrasil network, courtesy of our c ---- -## iPerf3 +## [iPerf3](https://iperf.fr) - `iperf3 -6 -c y.thingylabs.io` on [ygg.thingylabs.io](https://ygg.thingylabs.io/) - Link: 10 GBit From 7a377aff6cc586b1a5ec536f97f801d1e097203a Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 1 Aug 2019 22:55:50 +0100 Subject: [PATCH 16/95] Version 0.3.6 --- _posts/2019-08-03-release-v0-3-6.md | 114 ++++++++++++++++++++++++++++ changelog.md | 33 ++++++++ 2 files changed, 147 insertions(+) create mode 100644 _posts/2019-08-03-release-v0-3-6.md diff --git a/_posts/2019-08-03-release-v0-3-6.md b/_posts/2019-08-03-release-v0-3-6.md new file mode 100644 index 0000000..7d33341 --- /dev/null +++ b/_posts/2019-08-03-release-v0-3-6.md @@ -0,0 +1,114 @@ +--- +layout: post +title: "Version 0.3.6" +date: 2019-08-03 08:00:00 -0000 +author: Neil Alexander +--- + +### Version 0.3.6 + +It's been nearly five months since we released version 0.3.5 of Yggdrasil. In +that time we've seen the node count rise to over 400 nodes on the public network +at times (over 80% of which are running the latest released version) and we've +gained valuable insight to the kinds of challenges that our users have. We've +worked to fix a number of bugs and to improve Yggdrasil. + +In terms of lines of code changed, version 0.3.6 is the biggest release of +Yggdrasil to date, with several thousands of lines of code affected. It +represents a massive refactoring exercise in which we've broken up and +modularised the code, dividing core Yggdrasil functionality, TUN/TAP, admin +socket and multicast features into their own respective Go packages. + +### Fixes + +Most of the user-facing changes in this release are fairly minimal, however some +bugs have been corrected. A complete list is available in the [changelog](changelog.md). + +Highlights include peers now being added correctly even when one or more +configured peers are unavailable or unreachable. Multicast interfaces are also +being evaluated more frequently now, which can help if an interface becomes +available or goes down after Yggdrasil has already started. + +A number of bugs have been fixed in the TUN/TAP and IP-specific code, including +problems that affected ICMPv6 and Neighbour Discovery in TAP mode specifically. +This helps reliability on platforms where TAP mode is used more commonly, e.g. +on BSD platforms or on Windows, although this also improves TAP support on Linux +too. + +### Refactoring and API + +Around the previous release, it became obvious to us that our codebase was +turning into a monolith. We had pretty much all of the necessary behaviour in +a single `yggdrasil` package to run a single node, but this made our codebase +inflexible and difficult to maintain and extend. It also meant that Yggdrasil +was virtually impossible to integrate into other applications. + +Our refactoring efforts in version 0.3.6 mean that our codebase is now easier to +manage and to understand. It also includes the first taste of our API! The +API makes it possible to take the Yggdrasil core, drop it into your own Go +application and use the Yggdrasil network as a fully end-to-end encrypted and +distributed transport layer. We've also moved all of the IP-specific code into +the TUN/TAP module, which means that Yggdrasil's core now provides a completely +protocol-agnostic transport. + +Documentation on how to use the API to integrate Yggdrasil into your own +applications will follow soon—watch this space! In the meantime, `godoc` can be +used to examine our new API functions. + +Please note though that **API functions are not yet finalised and may be subject +to change** in future versions. Yggdrasil is still alpha-grade software at this +point so all of the usual warnings apply. + +### Platform Support + +We enjoy great support from our community in bringing and packaging Yggdrasil on +new platforms. Since the release of version 0.3.5, the following third-party +packages have cropped up, and we are very grateful to the maintainers: + +- A [new RPM build](https://copr.fedorainfracloud.org/coprs/leisteth/yggdrasil/) for Red Hat, Fedora, CentOS etc. +- An [AUR package](https://aur.archlinux.org/packages/yggdrasil-git/) for Arch Linux +- A [Void package](https://github.com/void-linux/void-packages/tree/master/srcpkgs/yggdrasil) for Void Linux +- A [MacPorts package](https://github.com/macports/macports-ports/blob/master/net/yggdrasil-go/Portfile) for macOS + +We expect that any third-party packages which have not yet been updated for +v0.3.6 will be updated soon! + +We are aware of a few outstanding issues with Windows, which are largely related +to one or two bugs in the [Water](https://github.com/songgao/water) library +which we use for TUN/TAP support. We are hoping to address these problems with +the maintainer of this library soon. Using Yggdrasil in router-only mode does +work as expected, but some bugs when using the TAP adapter still remain. In the +meantime, we'd certainly welcome any assistance in maintaining the Windows port +of Yggdrasil. + +The iOS build has been largely neglected due to API changes, although hopefully +a new TestFlight build for version 0.3.6 will be available before too long. + +### Upgrading + +We recommend that all Yggdrasil users always run the latest version of the code +wherever possible, so please upgrade as soon as it is convenient. New downloads +are available from our [Builds](builds.md) page and +[Neil](https://github.com/neilalexander)'s S3 repositories are up-to-date for +Debian and EdgeRouter installs. + +If you have installed through a package manager, you should be able to upgrade +in-place as soon as the new packages are available. On macOS, you can simply +install the new `.pkg` from the builds page over the top of the old one. On +Windows, and on any installation where the binary was installed by hand, you can +simply replace the `yggdrasil` and `yggdrasilctl` binaries with the newly +released builds. + +Building from source is simple if you have Git and Go 1.11 or later installed: +``` +git clone https://github.com/yggdrasil-network/yggdrasil-go +cd yggdrasil-go +./build +``` + +### Feedback + +We always welcome feedback, so please do feel free to join us either in our +[Matrix channel](https://riot.im/app/#/room/%23yggdrasil:matrix.org) or on IRC +in `#yggdrasil` on Freenode. You can also raise bug reports and issues in [our +GitHub repository](https://github.com/yggdrasil-network/yggdrasil-go/issues). diff --git a/changelog.md b/changelog.md index 5894598..3143d4d 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,39 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> +## [0.3.6] - 2019-08-03 +### Added +- Yggdrasil now has a public API with interfaces such as `yggdrasil.ConnDialer`, `yggdrasil.ConnListener` and `yggdrasil.Conn` for using Yggdrasil as a transport directly within applications +- Session gatekeeper functions, part of the API, which can be used to control whether to allow or reject incoming or outgoing sessions dynamically (compared to the previous fixed whitelist/blacklist approach) +- Support for logging to files or syslog (where supported) +- Platform defaults now include the ability to set sane defaults for multicast interfaces + +### Changed +- Following a massive refactoring exercise, Yggdrasil's codebase has now been broken out into modules +- Core node functionality in the `yggdrasil` package with a public API + - This allows Yggdrasil to be integrated directly into other applications and used as a transport + - IP-specific code has now been moved out of the core `yggdrasil` package, making Yggdrasil effectively protocol-agnostic +- Multicast peer discovery functionality is now in the `multicast` package +- Admin socket functionality is now in the `admin` package and uses the Yggdrasil public API +- TUN/TAP, ICMPv6 and all IP-specific functionality is now in the `tuntap` package +- `PPROF` debug output is now sent to `stderr` instead of `stdout` +- Node IPv6 addresses on macOS are now configured as `secured` +- Upstream dependency references have been updated, which includes a number of fixes in the Water library + +### Fixed +- Multicast discovery is no longer disabled if the nominated interfaces aren't available on the system yet, e.g. during boot +- Multicast interfaces are now re-evaluated more frequently so that Yggdrasil doesn't need to be restarted to use interfaces that have become available since startup +- Admin socket error cases are now handled better +- Various fixes in the TUN/TAP module, particularly surrounding Windows platform support +- Invalid keys will now cause the node to fail to start, rather than starting but silently not working as before +- Session MTUs are now always calculated correctly, in some cases they were incorrectly defaulting to 1280 before +- Multiple searches now don't take place for a single connection +- Concurrency bugs fixed +- Fixed a number of bugs in the ICMPv6 neighbor solicitation in the TUN/TAP code +- A case where peers weren't always added correctly if one or more peers were unreachable has been fixed +- Searches which include the local node are now handled correctly +- Lots of small bug tweaks and clean-ups throughout the codebase + ## [0.3.5] - 2019-03-13 ### Fixed - The `AllowedEncryptionPublicKeys` option has now been fixed to handle incoming connections properly and no longer blocks outgoing connections (this was broken in v0.3.4) From d258473b6c3f8b63bda47205361fc6f795611447 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 1 Aug 2019 23:04:39 +0100 Subject: [PATCH 17/95] Update version number on index.md --- index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.md b/index.md index 9f2ba45..5edb13d 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.5](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.6](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. From 52ee6dad123b5d5f7fdc70164bd33f88120b4b48 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sat, 3 Aug 2019 12:58:53 +0100 Subject: [PATCH 18/95] Update 2019-08-03-release-v0-3-6.md --- _posts/2019-08-03-release-v0-3-6.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_posts/2019-08-03-release-v0-3-6.md b/_posts/2019-08-03-release-v0-3-6.md index 7d33341..cfc5f4f 100644 --- a/_posts/2019-08-03-release-v0-3-6.md +++ b/_posts/2019-08-03-release-v0-3-6.md @@ -5,7 +5,7 @@ date: 2019-08-03 08:00:00 -0000 author: Neil Alexander --- -### Version 0.3.6 +### New release! It's been nearly five months since we released version 0.3.5 of Yggdrasil. In that time we've seen the node count rise to over 400 nodes on the public network @@ -52,8 +52,8 @@ the TUN/TAP module, which means that Yggdrasil's core now provides a completely protocol-agnostic transport. Documentation on how to use the API to integrate Yggdrasil into your own -applications will follow soon—watch this space! In the meantime, `godoc` can be -used to examine our new API functions. +applications will follow soon—watch this space! In the meantime, [`godoc` can be +used to examine our new API functions](https://godoc.org/github.com/yggdrasil-network/yggdrasil-go/src/yggdrasil). Please note though that **API functions are not yet finalised and may be subject to change** in future versions. Yggdrasil is still alpha-grade software at this From 6a884ed8854f083af9aadcc7868d22d53ac45c82 Mon Sep 17 00:00:00 2001 From: Arceliar Date: Sun, 4 Aug 2019 12:23:12 -0500 Subject: [PATCH 19/95] Update 2019-08-03-release-v0-3-6.md --- _posts/2019-08-03-release-v0-3-6.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-08-03-release-v0-3-6.md b/_posts/2019-08-03-release-v0-3-6.md index cfc5f4f..0b9cfa0 100644 --- a/_posts/2019-08-03-release-v0-3-6.md +++ b/_posts/2019-08-03-release-v0-3-6.md @@ -22,7 +22,7 @@ socket and multicast features into their own respective Go packages. ### Fixes Most of the user-facing changes in this release are fairly minimal, however some -bugs have been corrected. A complete list is available in the [changelog](changelog.md). +bugs have been corrected. A complete list is available in the [changelog](../changelog.md). Highlights include peers now being added correctly even when one or more configured peers are unavailable or unreachable. Multicast interfaces are also From e4704044a5813937f6282d44633ad3a2efa1cdc2 Mon Sep 17 00:00:00 2001 From: Arceliar Date: Sun, 4 Aug 2019 12:29:18 -0500 Subject: [PATCH 20/95] Update 2019-08-03-release-v0-3-6.md --- _posts/2019-08-03-release-v0-3-6.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-08-03-release-v0-3-6.md b/_posts/2019-08-03-release-v0-3-6.md index 0b9cfa0..a6e0c20 100644 --- a/_posts/2019-08-03-release-v0-3-6.md +++ b/_posts/2019-08-03-release-v0-3-6.md @@ -22,7 +22,7 @@ socket and multicast features into their own respective Go packages. ### Fixes Most of the user-facing changes in this release are fairly minimal, however some -bugs have been corrected. A complete list is available in the [changelog](../changelog.md). +bugs have been corrected. A complete list is available in the [changelog](/changelog.md). Highlights include peers now being added correctly even when one or more configured peers are unavailable or unreachable. Multicast interfaces are also From 34f0cd86738500b1167d2bb75e082192968e4769 Mon Sep 17 00:00:00 2001 From: Arceliar Date: Sun, 4 Aug 2019 12:30:56 -0500 Subject: [PATCH 21/95] Update 2019-08-03-release-v0-3-6.md --- _posts/2019-08-03-release-v0-3-6.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_posts/2019-08-03-release-v0-3-6.md b/_posts/2019-08-03-release-v0-3-6.md index a6e0c20..6248e08 100644 --- a/_posts/2019-08-03-release-v0-3-6.md +++ b/_posts/2019-08-03-release-v0-3-6.md @@ -22,7 +22,7 @@ socket and multicast features into their own respective Go packages. ### Fixes Most of the user-facing changes in this release are fairly minimal, however some -bugs have been corrected. A complete list is available in the [changelog](/changelog.md). +bugs have been corrected. A complete list is available in the [changelog](/changelog.html). Highlights include peers now being added correctly even when one or more configured peers are unavailable or unreachable. Multicast interfaces are also @@ -88,7 +88,7 @@ a new TestFlight build for version 0.3.6 will be available before too long. We recommend that all Yggdrasil users always run the latest version of the code wherever possible, so please upgrade as soon as it is convenient. New downloads -are available from our [Builds](builds.md) page and +are available from our [Builds](/builds.html) page and [Neil](https://github.com/neilalexander)'s S3 repositories are up-to-date for Debian and EdgeRouter installs. From f0e07fa31c7632ef724aaf96c6b1e6c7dba09d7a Mon Sep 17 00:00:00 2001 From: Jeff Date: Sun, 4 Aug 2019 16:02:39 -0400 Subject: [PATCH 22/95] Update services.md add another direct connect hub --- services.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/services.md b/services.md index 422f6f9..2d6a089 100644 --- a/services.md +++ b/services.md @@ -182,6 +182,8 @@ The following services are available on the Yggdrasil network, courtesy of our c - DC-hub by *whoami* `adc://[202:9877:2815:cd91:336:2a16:bfd9:7258]:1511` - Works with clients who support IPv6 and ADC. Ncdc or Airdcpp for example (eiskaltdc++ not support IPv6) +- bepis DC-hub run by *Jeff* `adc://[203:e0b0:ec08:4e1f:f004:19a9:577a:90ba]:1511` + ---- ## [iPerf3](https://iperf.fr) From cfa406ed84077495d2003548ab5fc9b934078490 Mon Sep 17 00:00:00 2001 From: passenger Date: Mon, 5 Aug 2019 19:03:40 +0200 Subject: [PATCH 23/95] add burger menu --- assets/css/style.scss | 56 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 2 deletions(-) diff --git a/assets/css/style.scss b/assets/css/style.scss index 5fdf139..ef2a413 100644 --- a/assets/css/style.scss +++ b/assets/css/style.scss @@ -22,15 +22,67 @@ div.wrapper section p a code { section { flex: auto; max-width: 100vw; + margin-right: 20px; + @media screen and (max-width: 767px) { + border: 0; + } } header { flex: initial; position: relative; - width: 240px !important; margin-right: 1em; + @media screen and (max-width: 767px) { + font-size: 0; + &:before { + content:'\2261'; + display: block; + position: absolute; + left: 0px; + font-size: 32px; + top: -30px; + } + img { + display: none; + } + > a { + position: absolute; + top: -23px; + left: 50px; + } + &:hover { + padding: 0; + font-size: initial; + > a { + pointer-events: none; + position: initial; + } + position: fixed; + height: 100%; + overflow: scroll; + background-color: white; + top: 0px; + width: 100%; + z-index: 100; + padding-top: 45px; + padding-bottom: 50px; + p { + a { + font-size: 24px; + padding-top: 2px; + padding-bottom: 2px; + display: block; + } + br { + display: none; + } + } + &:before { + display: none; + } + } + } } - pre { overflow-x: scroll; } From 76c0166dcdfe3f1e1a6d59b53ca2c7c858f20200 Mon Sep 17 00:00:00 2001 From: passenger Date: Tue, 6 Aug 2019 13:18:40 +0200 Subject: [PATCH 24/95] - fix scaling issues - add breakword - reset styling section - padding/margins --- assets/css/style.scss | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/assets/css/style.scss b/assets/css/style.scss index ef2a413..43cf8e1 100644 --- a/assets/css/style.scss +++ b/assets/css/style.scss @@ -23,8 +23,20 @@ section { flex: auto; max-width: 100vw; margin-right: 20px; - @media screen and (max-width: 767px) { + @media screen and (max-width: 768px) { + margin-top: 20px; border: 0; + float: none; + position: relative; + flex: initial; + word-break: break-word; + table { + thead { + th { + min-width: 90px; + } + } + } } } @@ -32,15 +44,20 @@ header { flex: initial; position: relative; margin-right: 1em; - @media screen and (max-width: 767px) { + @media screen and (max-width: 768px) { + > p { + padding-right: 20%; + } font-size: 0; + margin: 0; + padding: 0; &:before { content:'\2261'; display: block; position: absolute; left: 0px; - font-size: 32px; - top: -30px; + font-size: 40px; + top: -38px; } img { display: none; @@ -50,6 +67,11 @@ header { top: -23px; left: 50px; } + a { + &:hover { + font-weight: initial; + } + } &:hover { padding: 0; font-size: initial; @@ -59,13 +81,17 @@ header { } position: fixed; height: 100%; - overflow: scroll; + overflow-y: scroll; background-color: white; top: 0px; + left: 0; + right: 0; + bottom: 0; width: 100%; z-index: 100; - padding-top: 45px; + padding-top: 20px; padding-bottom: 50px; + padding-left: 20px; p { a { font-size: 24px; From a1cc2194fdd2337fafe8a26fd1fb961aa49c8082 Mon Sep 17 00:00:00 2001 From: Eninspace <38405119+Eninspace@users.noreply.github.com> Date: Wed, 7 Aug 2019 16:08:28 +0300 Subject: [PATCH 25/95] Create installation-linux-openwrt.md --- installation-linux-openwrt.md | 46 +++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 installation-linux-openwrt.md diff --git a/installation-linux-openwrt.md b/installation-linux-openwrt.md new file mode 100644 index 0000000..d765a89 --- /dev/null +++ b/installation-linux-openwrt.md @@ -0,0 +1,46 @@ +--- +tags: dontlink +sitemap: true +--- + +# Installing on OpenWrt 19.07 + +Yggdrasil is supported on the OpenWrt based router. + + +Perform installation steps over SSH by connecting to the Router as the +`root` user, e.g. `ssh root@192.168.1.1`, or another admin-level user if +configured. + +## Package install from repository + +Install the `yggdrasil` package: +``` +opkg update +opkg install yggdrasil +``` + +Yggdrasil can then be updated in the future by updating the packages: +``` +opkg update +opkg upgrade yggdrasil +``` + +## Installing on OpenWrt 18.06 and below + +Officially, there is no package support on the OpenWrt 18.06, so you may have any problems. +``` +opkg update +opkg install http://downloads.openwrt.org/releases/packages-19.07/xxxxxx_xxx/packages/yggdrasil_x.x.x-x_xxxxxx_xxx.ipk +``` +Where xxxxxx_xxx your architecture, and yggdrasil_x.x.x-x_xxxxxx_xxx.ipk name of package +For example +``` +opkg install http://downloads.openwrt.org/releases/packages-19.07/mipsel_24kc/packages/yggdrasil_0.3.5-4_mipsel_24kc.ipk +``` +This method does not guarantee a successful installation + +## After installation + +Read the [OpenWrt platform page](platform-openwrt.md) page for information +on how to configure Yggdrasil and guidance on modifying the configuration. From ad35c566b2885c3c8578f59472e2713a35c1e0cc Mon Sep 17 00:00:00 2001 From: Mihail Podivilov Date: Fri, 9 Aug 2019 15:37:57 +0300 Subject: [PATCH 26/95] Update services.md --- services.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/services.md b/services.md index 2d6a089..5af1fb1 100644 --- a/services.md +++ b/services.md @@ -150,6 +150,8 @@ The following services are available on the Yggdrasil network, courtesy of our c - Will ask 1.1.1.1 over TLS, recursive resolving currently disabled due to inferior speed. - `200:a120:8cf8:2ad5:7509:7341:c861:34aa` port 53 hosted by [sin](https://2f30.org) - Supports clearnet resolution of standard DNS domains via root lookups + - `200:d0c4:68ee:e87b:c206:67b8:5fa5:d4be` port 53, hosted by [Medium](https://github.com/medium-isp) + - Internal Yggdrasil services resolver ---- From 5232162171ebeae9a0468b6ae4d2de25020b3d2a Mon Sep 17 00:00:00 2001 From: Eninspace <38405119+Eninspace@users.noreply.github.com> Date: Tue, 13 Aug 2019 16:03:43 +0300 Subject: [PATCH 27/95] Update installation-linux-openwrt.md --- installation-linux-openwrt.md | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/installation-linux-openwrt.md b/installation-linux-openwrt.md index d765a89..ee6c839 100644 --- a/installation-linux-openwrt.md +++ b/installation-linux-openwrt.md @@ -26,20 +26,6 @@ opkg update opkg upgrade yggdrasil ``` -## Installing on OpenWrt 18.06 and below - -Officially, there is no package support on the OpenWrt 18.06, so you may have any problems. -``` -opkg update -opkg install http://downloads.openwrt.org/releases/packages-19.07/xxxxxx_xxx/packages/yggdrasil_x.x.x-x_xxxxxx_xxx.ipk -``` -Where xxxxxx_xxx your architecture, and yggdrasil_x.x.x-x_xxxxxx_xxx.ipk name of package -For example -``` -opkg install http://downloads.openwrt.org/releases/packages-19.07/mipsel_24kc/packages/yggdrasil_0.3.5-4_mipsel_24kc.ipk -``` -This method does not guarantee a successful installation - ## After installation Read the [OpenWrt platform page](platform-openwrt.md) page for information From 8493f7d86ae2a565ee93178cde972d95cbcf3726 Mon Sep 17 00:00:00 2001 From: whoami Date: Sun, 18 Aug 2019 02:38:20 +0300 Subject: [PATCH 28/95] service.md: remove whoami DC-hub forgot to delete. This hub does not work because I no longer pay for VPS --- services.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/services.md b/services.md index 2d6a089..a261d95 100644 --- a/services.md +++ b/services.md @@ -179,9 +179,6 @@ The following services are available on the Yggdrasil network, courtesy of our c ## Direct Connect -- DC-hub by *whoami* `adc://[202:9877:2815:cd91:336:2a16:bfd9:7258]:1511` - - Works with clients who support IPv6 and ADC. Ncdc or Airdcpp for example (eiskaltdc++ not support IPv6) - - bepis DC-hub run by *Jeff* `adc://[203:e0b0:ec08:4e1f:f004:19a9:577a:90ba]:1511` ---- From b58db34c001974968afa5ca2477a1d85ed63db57 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sun, 18 Aug 2019 11:25:57 +0100 Subject: [PATCH 29/95] Update changelog.md --- changelog.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/changelog.md b/changelog.md index 3143d4d..b7775af 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,25 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> +## [0.3.7] - 2019-08-18 +### Changed +- The switch should now forward packets along a single path more consistently in cases where congestion is low and multiple equal-length paths exist, which should improve stability and result in fewer out-of-order packets +- Sessions should now be more tolerant of out-of-order packets, by replacing a bitmask with a variable sized heap+map structure to track recently received nonces, which should reduce the number of packets dropped due to reordering when multiple paths are used or multiple independent flows are transmitted through the same session +- The admin socket can no longer return a dotfile representation of the known parts of the network, this could be rebuilt by clients using information from `getSwitchPeers`,`getDHT` and `getSessions` + +### Fixed +- A number of significant performance regressions introduced in version 0.3.6 have been fixed, resulting in better performance +- Flow labels are now used to prioritise traffic flows again correctly +- In low-traffic scenarios where there are multiple peerings between a pair of nodes, Yggdrasil now prefers the most active peering instead of the least active, helping to reduce packet reordering +- The `Listen` statement, when configured as a string rather than an array, will now be parsed correctly +- The admin socket now returns `coords` as a correct array of unsigned 64-bit integers, rather than the internal representation +- The admin socket now returns `box_pub_key` in string format again +- Sessions no longer leak/block when no listener (e.g. TUN/TAP) is configured +- Incoming session connections no longer block when a session already exists, which results in less leaked goroutines +- Flooded sessions will no longer block other sessions +- Searches are now cleaned up properly and a couple of edge-cases with duplicate searches have been fixed +- A number of minor allocation and pointer fixes + ## [0.3.6] - 2019-08-03 ### Added - Yggdrasil now has a public API with interfaces such as `yggdrasil.ConnDialer`, `yggdrasil.ConnListener` and `yggdrasil.Conn` for using Yggdrasil as a transport directly within applications From 6df74fd50a844affe721c21df6daba5892c40a1c Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sun, 18 Aug 2019 11:26:09 +0100 Subject: [PATCH 30/95] Update index.md --- index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.md b/index.md index 5edb13d..cd7503e 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.6](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.7](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. From 29a31cd9ebe7c4687f3c6e62295aa158ccede76a Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 19 Aug 2019 20:57:03 +0100 Subject: [PATCH 31/95] Add AWDL post --- _posts/2019-08-19-awdl.md | 227 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 _posts/2019-08-19-awdl.md diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md new file mode 100644 index 0000000..4e72ee8 --- /dev/null +++ b/_posts/2019-08-19-awdl.md @@ -0,0 +1,227 @@ +--- +layout: post +title: "Meshing using Apple Wireless Device Link (AWDL)" +date: 2019-08-19 08:00:00 -0000 +author: Neil Alexander +--- + +### About AWDL + +I was mostly prompted to write this post in response to a [Hacker News +thread](https://news.ycombinator.com/item?id=20735462) recently, which announced +the release of an open-source implementation of the Apple Wireless Device Link +(AWDL) protocol. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay +and some other Apple wireless technologies. + +With a few exceptions, most wireless networks in the world operate in +"infrastructure mode" which is where a wireless access point serves one or more +wireless clients. Think of your Wi-Fi at home, at work or in a coffee shop. +However, as implied by the name, reliable and usable infrastructure Wi-Fi is +often only available in certain physical locations with "good infrastructure". +If you wanted to connect some devices together anywhere not served by an +infrastructure Wi-Fi network, or in a location where you can't suddenly plug in +a wireless access point, you may not have many options (Bluetooth aside). + +AWDL is designed to avoid this problem by extending the 802.11 wireless standard +to allow client devices to communicate directly with each other, without the +help of the central wireless access point. You can walk out into a field with a +couple of iPhones or Macs and they can use AWDL to discover each other and +exchange data, peer-to-peer. Even better is that nearby devices that are +connected to different infrastructure Wi-Fi networks can still communicate with +each other using AWDL! + +### The science + +Normally, when connected to a wireless access point, wireless clients remain +locked to the specific radio channel that the AP is using. AWDL works by +instructing the wireless adapter in the device to "hop" between channels so that +it can not only remain connected to the wireless access point, but can also +listen to other nearby devices. + +Devices announce their presence and information about their services on a +"social channel" for other devices to hear, effectively creating peer-to-peer +service discovery. Once two devices have decided that they want to communicate +directly, they agree to jump to another channel for real data exchange so that +they don't interrupt existing Wi-Fi networks or, indeed, the social channel. +These "hops" between wireless channels happen so quickly that there's very +little disruption to what the user is doing with their Wi-Fi connection already +(except for some minor wireless performance degradation - to be covered later). + +### Mesh opportunities + +Yggdrasil is designed to create a mesh network automatically out of +interconnected nodes - the idea being that all nodes can route to all other +nodes on the mesh network by routing through other nodes. + +Today, many of these connections happen between nodes across the Internet, since +the community is still relatively small and geographically dispersed. A node +joining the Yggdrasil network needs to only peer with a single device that is +already connected to the wider network in order to participate in the +fully-routable mesh. + +However, it's not the goal of Yggdrasil to remain something that we just toy +with over the Internet. We want to build a protocol that can scale globally and +work ad-hoc, even in places where infrastructure might not be particularly +strong otherwise. We think that one of Yggdrasil's greatest strengths is that it +is very close to zero-configuration, beyond giving it a very small number of +configuration options, and it should scale well too in principle. + +Yggdrasil can already discover potential peers on the same network segment by +using multicast service discovery, which sounds a lot like what AWDL does on the +social channel. You can configure which interfaces Yggdrasil beacons on with the `MulticastInterfaces` configuration directive. + +I wanted to know if we could blend the two so that Yggdrasil could automatically +discover other nearby devices and initiate peering connections with them using +AWDL. + +### Getting started + +Macs are a good target for developing and testing AWDL-aware applications as +AWDL is exposed to userspace through a network adapter called `awdl0`. It sits +there with a link-local IPv6 address, you can run `tcpdump` or Wireshark on it +to listen to AWDL traffic and you can even ping multicast group addresses on the +interface and get responses from other nearby devices, e.g. using `ping6 +ff02::1%awdl0`! However, Apple devices don't always keep AWDL alive and +listening all of the time. + +On macOS, the AWDL driver is only woken up when either AirDrop is being +actively used in Finder, or where a `NetService` has been created (usually +through Objective-C or Swift) which requests peer-to-peer networking. AWDL is +normally kept alive long enough to satisfy connectivity for these sessions and +then will be sent back to sleep after a period of idleness. + +On iOS, the story is somewhat similar to above, except that AWDL is often woken +up as soon as the device is unlocked if AirDrop is enabled. The `NetService` API +otherwise functions the same way. + +tvOS is the outlier in that it seems to wake up and listen to AWDL randomly, +even when the device is otherwise asleep, presumably because it is advertising +the ability to receive incoming AirPlay sessions to nearby devices. + +From a user perspective, the `awdl0` interface looks entirely unremarkable. It +behaves largely like any other ethernet interface, carrying regular IPv6 +traffic. In the background it's a bit more complicated, as the AWDL driver +performs traffic filtering for security reasons, namely, to stop someone sat +next to you in the airport from browsing your file shares. Namely, regular +listening sockets won't accept connections over AWDL unless a specific socket +option was configured on the socket before it started listening. + +Multicast traffic, however, does largely get passed through the filter +untouched. Bingo. + +### Waking up AWDL + +The `NetService` API is effectively a wrapper around multicast DNS-SD, which in +Apple's colourful language, is affectionately known as Bonjour. The API has the +added benefit of being able to tell the operating system to wake up the AWDL +driver pretty much on demand on behalf of "peer-to-peer" services. + +So all we would need to do to wake up AWDL is to call the `NetService` API, +publish a service that requests peer-to-peer functionality and let the operating +system do the hard work for us. Yggdrasil, being written in Go, didn't have any +concept of `NetService` but thankfully we were able to use Cgo to do this +instead. + +We wrote a Cgo function which calls the NetService API and advertises our new +fake service, `_yggdrasil._tcp`, which causes the operating system to wake up +the AWDL driver. Amazingly this worked. + +Yggdrasil doesn't actually use DNS-SD - we currently use a custom-formatted +multicast beacon on a different multicast group. It is planned to eventually +migrate to something more standard, like DNS-SD, for service discovery. However, +in this instance, registering a fake DNS-SD service was just enough to wake up +AWDL. + +### Peering automatically + +Once the driver is active, the regular Yggdrasil multicast beacons on the +`ff02::114` multicast group address seem to be passed through to the driver +normally and the Yggdrasil nodes running on each machine start to hear each +other's calls. + +The only thing that remained to be done was to configure the sockets with the +aforementioned socket option to allow them to communicate over the AWDL +interface. This socket option is called `SO_RECV_ANYIF` and is defined in +`sys/socket.h` on Darwin as `0x1104`. + +We configure the socket option on our TCP peering socket: +``` +err = unix.SetsockoptInt(int(fd), syscall.SOL_SOCKET, 0x1104, 1) +if err != nil { + ... +} +``` + +Now that the Yggdrasil nodes can hear each other's advertisements over the +`awdl0` interface, the regular automatic peering process kicks in and a TCP +session is opened between the two devices, creating a peering. The net result? +AWDL peerings! +``` +$ sudo yggdrasilctl getSwitchPeers + bytes_recvd bytes_sent coords endpoint ip port proto +1 304430 246994 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%en0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 1 tcp +2 1176278 878133 [3 5 5] fe80::xxxx:xxxx:xxxx:xxxx%en1 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 2 tcp +3 244278 313907 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%awdl0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 3 tcp +``` + +To further cement the experiment, we can actually disconnect the two devices +from each other, or connect to different Wi-Fi networks automatically, and the +peering over the `awdl0` interface still continues to function! + +An `iperf3` test over Yggdrasil using the new AWDL link looks fairly good - the +devices are sat next to each other: +``` +[ ID] Interval Transfer Bandwidth +[ 5] 0.00-1.00 sec 15.4 MBytes 129 Mbits/sec +[ 5] 1.00-2.00 sec 16.9 MBytes 141 Mbits/sec +[ 5] 2.00-3.00 sec 15.9 MBytes 133 Mbits/sec +[ 5] 3.00-4.00 sec 17.6 MBytes 147 Mbits/sec +[ 5] 4.00-5.00 sec 16.8 MBytes 141 Mbits/sec +[ 5] 5.00-6.00 sec 16.2 MBytes 136 Mbits/sec +[ 5] 6.00-7.00 sec 12.5 MBytes 105 Mbits/sec +[ 5] 7.00-8.00 sec 12.7 MBytes 106 Mbits/sec +[ 5] 8.00-9.00 sec 14.9 MBytes 125 Mbits/sec +[ 5] 9.00-10.00 sec 13.5 MBytes 113 Mbits/sec +``` + +### Observations and iOS + +As the `iperf3` test above shows, the link performance is actually quite good! +It routinely exceeds 100mbps, although this is between only two devices. I have +not been able to test this with Yggdrasil nodes running over AWDL in any +particular density due to only having a limited number of Macs to hand. + +One thing that I did notice though is that, while AWDL is active, my wireless +connection to my home Wi-Fi network does reduce in speed somewhat. This is to be +expected, given that the wireless chipset is hopping between channels rather +than spending all of it's time on a single channel. + +Sadly we weren't able to reproduce this test using iOS Testflight builds of +Yggdrasil. On iOS, we implement Yggdrasil as a VPN service which is subject to a +number of probably reasonable restrictions imposed by the OS, which presumably +exist to stop VPN extensions from spying on you. + +We were able to create a `NetService` from within the VPN extension and the +service beacons were advertised as expected, however, we weren't able to +initiate any other kind of connections over the `awdl0` interface. After a chat +with an engineer at Apple, it turns out that the `awdl0` interface isn't scoped +for use within a VPN extension, thus squashing our hopes and dreams of being +able to sprinkle this kind of magic onto our iOS port of Yggdrasil. We have a +feature request radar open with Apple in the hope that they may be able to +change this restriction in the future. + +But we were able to get this to work on macOS and that, itself, is quite +awesome. + +### Conclusion + +Yggdrasil doesn't enable AWDL by default because of the reduction in wireless +performance that AWDL being active can cause. Therefore, to enable AWDL peering, +you must add the `awdl0` interface specifically into the `MulticastInterfaces` +configuration option in `yggdrasil.conf`. However, we do have working support +for connecting Macs together and meshing automatically using AWDL, and you can +enable it very easily if you wish to experiment! + +We'd love to hear if you are peering Yggdrasil nodes using AWDL, or have +performed any more extensive tests of how it performs in real-world scenarios - +join us on our [Matrix channel](https://matrix.to/#/#yggdrasil:matrix.org)! From ec0da4738761139313ffd6c0c2960e80ba22d5ec Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 19 Aug 2019 21:00:20 +0100 Subject: [PATCH 32/95] Update AWDL post --- _posts/2019-08-19-awdl.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md index 4e72ee8..6eb91fe 100644 --- a/_posts/2019-08-19-awdl.md +++ b/_posts/2019-08-19-awdl.md @@ -5,7 +5,7 @@ date: 2019-08-19 08:00:00 -0000 author: Neil Alexander --- -### About AWDL +### Wireless without borders I was mostly prompted to write this post in response to a [Hacker News thread](https://news.ycombinator.com/item?id=20735462) recently, which announced @@ -159,9 +159,7 @@ AWDL peerings! ``` $ sudo yggdrasilctl getSwitchPeers bytes_recvd bytes_sent coords endpoint ip port proto -1 304430 246994 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%en0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 1 tcp -2 1176278 878133 [3 5 5] fe80::xxxx:xxxx:xxxx:xxxx%en1 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 2 tcp -3 244278 313907 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%awdl0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 3 tcp +1 244278 313907 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%awdl0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 1 tcp ``` To further cement the experiment, we can actually disconnect the two devices From 3877e65ffb6ed0759f66af3adc3c6c2104dd72d6 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 19 Aug 2019 21:15:32 +0100 Subject: [PATCH 33/95] Fix name --- _posts/2019-08-19-awdl.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md index 6eb91fe..15b82e9 100644 --- a/_posts/2019-08-19-awdl.md +++ b/_posts/2019-08-19-awdl.md @@ -1,6 +1,6 @@ --- layout: post -title: "Meshing using Apple Wireless Device Link (AWDL)" +title: "Meshing using Apple Wireless Direct Link (AWDL)" date: 2019-08-19 08:00:00 -0000 author: Neil Alexander --- @@ -9,7 +9,7 @@ author: Neil Alexander I was mostly prompted to write this post in response to a [Hacker News thread](https://news.ycombinator.com/item?id=20735462) recently, which announced -the release of an open-source implementation of the Apple Wireless Device Link +the release of an open-source implementation of the Apple Wireless Direct Link (AWDL) protocol. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless technologies. From c63712229a28ef778dd7b32419bc9017e190d9d6 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 19 Aug 2019 21:17:16 +0100 Subject: [PATCH 34/95] Update AWDL post --- _posts/2019-08-19-awdl.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md index 15b82e9..f16f2fb 100644 --- a/_posts/2019-08-19-awdl.md +++ b/_posts/2019-08-19-awdl.md @@ -11,7 +11,8 @@ I was mostly prompted to write this post in response to a [Hacker News thread](https://news.ycombinator.com/item?id=20735462) recently, which announced the release of an open-source implementation of the Apple Wireless Direct Link (AWDL) protocol. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay -and some other Apple wireless technologies. +and some other Apple wireless technologies. Even though everything covered in +this post was done some time ago, I have never spent the time documenting it. With a few exceptions, most wireless networks in the world operate in "infrastructure mode" which is where a wireless access point serves one or more From 716d07023b7ff07d97399d3c1fbcbd9239d26878 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 19 Aug 2019 21:18:29 +0100 Subject: [PATCH 35/95] Update AWDL post --- _posts/2019-08-19-awdl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md index f16f2fb..37b2ef2 100644 --- a/_posts/2019-08-19-awdl.md +++ b/_posts/2019-08-19-awdl.md @@ -12,7 +12,7 @@ thread](https://news.ycombinator.com/item?id=20735462) recently, which announced the release of an open-source implementation of the Apple Wireless Direct Link (AWDL) protocol. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless technologies. Even though everything covered in -this post was done some time ago, I have never spent the time documenting it. +this post was done some time ago, I have never spent the time to document it. With a few exceptions, most wireless networks in the world operate in "infrastructure mode" which is where a wireless access point serves one or more From 1972585725c8031754d8e8e24c505e9738d45277 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 19 Aug 2019 22:42:00 +0100 Subject: [PATCH 36/95] Update 2019-08-19-awdl.md --- _posts/2019-08-19-awdl.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md index 37b2ef2..327c169 100644 --- a/_posts/2019-08-19-awdl.md +++ b/_posts/2019-08-19-awdl.md @@ -103,9 +103,9 @@ From a user perspective, the `awdl0` interface looks entirely unremarkable. It behaves largely like any other ethernet interface, carrying regular IPv6 traffic. In the background it's a bit more complicated, as the AWDL driver performs traffic filtering for security reasons, namely, to stop someone sat -next to you in the airport from browsing your file shares. Namely, regular -listening sockets won't accept connections over AWDL unless a specific socket -option was configured on the socket before it started listening. +next to you in the airport from browsing your file shares. Regular listening +sockets won't accept connections over AWDL unless a specific socket option was +configured on the socket before it started listening. Multicast traffic, however, does largely get passed through the filter untouched. Bingo. From 5dd68d68ac08fb7d3e83813c683bff8ee65c82e2 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 21 Aug 2019 18:21:50 +0100 Subject: [PATCH 37/95] Version 0.3.8 --- changelog.md | 19 +++++++++++++++++++ index.md | 2 +- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/changelog.md b/changelog.md index b7775af..b0be64d 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,25 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> +## [0.3.8] - 2019-08-21 +### Changed +- Yggdrasil can now send multiple packets from the switch at once, which results in improved throughput with smaller packets or lower MTUs +- Performance has been slightly improved by not allocating cancellations where not necessary +- Crypto-key routing options have been renamed for clarity + - `IPv4Sources` is now named `IPv4LocalSubnets` + - `IPv6Sources` is now named `IPv6LocalSubnets` + - `IPv4Destinations` is now named `IPv4RemoteSubnets` + - `IPv6Destinations` is now named `IPv6RemoteSubnets` + - The old option names will continue to be accepted by the configuration parser for now but may not be indefinitely +- When presented with multiple paths between two nodes, the switch now prefers the most recently used port when possible instead of the least recently used, helping to reduce packet reordering +- New nonce tracking should help to reduce the number of packets dropped as a result of multiple/aggregate paths or congestion control in the switch + +### Fixed +- **Security vulnerability**: Address verification was not strict enough, which could result in a malicious session sending traffic with unexpected or spoofed source or destination addresses which Yggdrasil could fail to reject + - Versions `0.3.6` and `0.3.7` are vulnerable - users of these versions should upgrade as soon as possible + - Versions `0.3.5` and earlier are not affected +- A deadlock was fixed in the session code which could result in Yggdrasil failing to pass traffic after some time + ## [0.3.7] - 2019-08-18 ### Changed - The switch should now forward packets along a single path more consistently in cases where congestion is low and multiple equal-length paths exist, which should improve stability and result in fewer out-of-order packets diff --git a/index.md b/index.md index cd7503e..ed5d292 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.7](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.8](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. From b281da87ac0e465e5feab9ef808ec0aed7cedc44 Mon Sep 17 00:00:00 2001 From: "J. B" <53562520+user2kcd@users.noreply.github.com> Date: Mon, 26 Aug 2019 17:06:04 +0500 Subject: [PATCH 38/95] Update services.md --- services.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/services.md b/services.md index 657fe95..0684cbb 100644 --- a/services.md +++ b/services.md @@ -152,7 +152,8 @@ The following services are available on the Yggdrasil network, courtesy of our c - Supports clearnet resolution of standard DNS domains via root lookups - `200:d0c4:68ee:e87b:c206:67b8:5fa5:d4be` port 53, hosted by [Medium](https://github.com/medium-isp) - Internal Yggdrasil services resolver - + - `200:5bf2:a7a5:27c7:54b8:6669:eb74:1813` port 53, hosted by [User2k](https://user2k.eu) + - Medium DNS + OpenDNS + Google DNS + CloudFlare DNS ---- ## ZeroNet From 10753b4746d357d375c49e6af31682f962a0df1c Mon Sep 17 00:00:00 2001 From: Arceliar Date: Sun, 1 Sep 2019 15:46:44 -0500 Subject: [PATCH 39/95] blog post about actor model changes --- _posts/2019-09-01-actors.md | 171 ++++++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 _posts/2019-09-01-actors.md diff --git a/_posts/2019-09-01-actors.md b/_posts/2019-09-01-actors.md new file mode 100644 index 0000000..abfae37 --- /dev/null +++ b/_posts/2019-09-01-actors.md @@ -0,0 +1,171 @@ +--- +layout: post +title: "Acting out" +date: 2019-09-01 21:00:00 +0000 +author: Arceliar +--- + +### Overture + +We've recently rewritten much of Yggdrasil's internals to change from Go's native [communicating sequential processes](https://en.wikipedia.org/wiki/Communicating_sequential_processes) (goroutine+channel) style to using an asynchronous [actor model](https://en.wikipedia.org/wiki/Actor_model) approach to concurrency. While this change should be invisible to the average user, it dramatically changes what we developers need to think about when working on the code. I thought it would be useful to explain a little about the motivation for rewriting things this way, and what the consequences are. + +Caution: theatre puns and references throughout, because `Actor`s. + +### Exposition + +Yggdrasil is written in the Go programming language. Go makes it easy to start a function running concurrently, and gives developers the tools they need to make concurrently executing functions communicate, but it's not always easy to use them correctly. To be clear, the things I'm about to rant about are all fixable. Working around them is a normal thing to do in Go. More importantly, it's a case where doing things the obvious way (which is sometimes even safe in isolation) leads to *wrong* behavior in a larger program. I prefer models where the obvious thing is still correct, and non-obvious things are only needed as a performance optimization. + +#### Composition + +There's a common pattern that has emerged many times in the Yggdrasil code base. We'll have a `struct` with some mutable fields that need reading or updating, such as information about a particular cryptographic session, or the switch's table of idle peers and buffered traffic. Since shared mutable state is hard, and Go is all about "[Share Memory By Communicating](https://blog.golang.org/share-memory-by-communicating)", we'll have packets get passed to a dedicated worker goroutine that "owns" that particular `struct`. The worker uses information from the packet and the owned `struct` to do whatever it is needs to do, updates these things accordingly, and passes the packet along to the next goroutine in the pipeline. + +This often results in a "`for select`" pattern, where goroutines sit in an infinite `for` loop and `select` on several channels, to wait for packets to process or various types of signals from other goroutines. There are a few ways around it (with heavy use of `reflect` or `chan interface{}`, for example), but in most cases, every `select` statement needs to fully enumerate every behavior that the goroutine may need to engage in at that point in the code. If there's a common set of `case`s that always need to be handled, and then a few exceptional `case`s that may or may not matter (possibly when the associated `struct`s the workers are using are similar but not exactly the same types, or as the state of a `struct`'s fields change), then that typically involves multiple `select` statements with only the addition or modification of one or two `case`s. + +Go embraces composition in its type system, but `select` statements (and channel operations in general) make execution resistant to composition. + +#### Deadlocks + +The "`for select`" pattern is safe, as far as I know, if the flow of messages through the program form a directed acyclic graph. However, in our case, cycles emerge if we try to handle things in the obvious way. For example, a cryptographic session needs to somehow get outbound encrypted traffic to the switch, but incoming encrypted traffic also needs to make it from the switch to the sessions for decryption (via the router, which is responsible for, among other things, identify which session is associated with the traffic). + +When cycles of goroutines naively pass messages over channels, deadlocks are all but inevitable. There are a few ways to address this, but they're not always appropriate. Ideally, we would change the design to remove cycles, but this is not always possible, and may require significant changes to the workflow in cases where it is possible. In practice, what we'd actually do is either buffer messages (having some dedicated reader goroutine to take the message, add it to a slice, and then pass it to the real destination ASAP) or drop messages entirely (with a `select` statement that aborts and does cleanup in a `default` case, or by having a dedicated reader that drops messages more intelligently, such as from the front of the queue, under the assumption that older messages are less useful). + +#### Leaks + +Typically, when a goroutine is started, it continues to run until either the function returns or the program exits. For this reason, if a goroutine executes any statements which can block (such as a channel operation), it's important to include some `case` which signals that it's time to return. Forgetting to do this can result in goroutine leaks. [Never start a goroutine without knowing how it will stop](https://dave.cheney.net/2016/12/22/never-start-a-goroutine-without-knowing-how-it-will-stop), or so the experts say. + +This is sometimes harder than it needs to be. To be blunt, the single producer N consumer cases are fine, you just close the channel and have all the consumers take this as a signal to exit. Anything involving multiple producers requires some sort of signaling to indicate that all producers have exited. Since you're using a channel already, the obvious option is a `select` statement with another channel that closes to signal shutdown, and then something like e.g. a `[sync.WaitGroup](https://golang.org/pkg/sync/#WaitGroup)` to wait for all producers to exit before closing the channel. Until your number of producers needs to change at runtime, and you realize that this races if you start to `Wait` before `Add`ing everything to the group, so you need to implement a custom counter, and be careful that additions and subtractions can also race and cause it to shut down early. And have fun solving it, because with how much `select` resists composition and code reuse, you're going to be implementing the same patterns over, and over, and over, and over... + +It's not that this is some impossible problem to solve, it's just that Go's take on the [CSP](https://en.wikipedia.org/wiki/Communicating_sequential_processes), combined with the rest of the tools the language gives you, makes it easy and concise to run thing the *wrong* way, and leads to comparatively complex and delicate code when trying to run it the right way. At least, that's my personal view of it based on my experience so far, but it probably varies some based on the problem the code is trying to solve. + +### Rising action + +The [actor model](https://en.wikipedia.org/wiki/Actor_model) is another programming paradigm that embraces concurrency with a "share memory by communicating" philosophy. + +For our purposes, an actor is basically a data type with a few special properties: +1. It has an inbox where messages to the actor are placed. +2. It has an associated unit of execution, such as a thread, which processes messages from the inbox one at a time. +3. Rather than exposing ordinary functions for other code to call, the actor exposes *behaviors*. A behavior is a function which has no return value, and is executed only for its side effects. When an actor `A` calls a behavior of an actor `B`, what really happens is that `A` places a message in `B`'s inbox, and `B` processes that message by executing some code. + +Different implementations differ on details after that, such as what order messages are processed in, if actors are allowed to wait for a particular type of message before continuing, whether actors run locally or are distributed across a cluster, etc., but they tend to all include some version of the broad strokes above. + +### Turing point + + + +I'm particularly fond of the [pony](https://ponylang.io) programming language's take on the actor model. I really can't being to say enough nice things about their approach, and fully describing it is beyond the scope of this blog post, but if you come out of here with an interest in the actor model, then I highly recommend checking out that language. Maybe watch a few of the talks from the developers that have been posted to youtube, or read their papers about what is *easily* the most promising approach to garbage collection I've ever come across. + +Anyway, I don't actually work on anything written in pony, but I like their version of the actor model so much that I decided to see if I could trick Go's runtime into faking it. The result is [`phony`](https://github.com/Arceliar/yggdrasil-go), which manages to do most of what I want in under 70 lines of code. When we write code using this asynchronous message passing style, instead of ordinary goroutines+channels, the implications are pretty significant: + +1. There are no deadlocks. Message sends always succeed, and are quite fast (it doesn't even require [CAS](https://en.wikipedia.org/wiki/Compare-and-swap) instructions in the normal case). +2. Inbox sizes stay small due to backpressure: if the sender sees that the receiver's inbox has too many pending messages, it will schedule itself to stop at some deadlock-free safe point in the future, to wait until the receiver signals that it's handled the message. +3. `Actor`s are *shockingly* lightweight: on a modern 64-bit processor, an idle `Actor`'s only resources are 24 bytes for an empty `Inbox`, some of which is padding that may not apply if embedded into a struct. In particular, an idle `Actor` with an empty `Inbox` has no associated goroutine, so it requires no stack. +4. The lack of a goroutine also means that idle `Actor`s, even cycles of `Actor`s, can be garbage collected automatically. +5. Any `struct` that embeds an `Inbox` satisfies the `Actor` interface. Since `Actor`s encapsulate their own unit of execution, it means the range of behaviors that unit of execution can engage in are encoded into the type system and can even be abstracted through `interface` types. In my opinion, the resulting code is cleaner, easier to read and understand, and far easier to reuse or extend than the `for select` pattern from goroutine+channel use. + +### Falling action + +I'm happy enough with the current state of `phony` that I decided to start migrating the `yggdrasil-go` code base to use it. This is still work in progress (there are some non-`Actor` goroutines around the edges of the code, mostly in main `Accept` loops and that sort of thing), but the hot paths are now `Actor` based. + +Most of this was done in a weekend and came together with surprisingly little pain. I had exactly 2 crashes the entire time (1 accidental `nil` pointer deference and 1 legitimate bug I needed to fix in `phony`), and more importantly, 0 deadlocks. Most things just worked as intended the first time they compiled. There were a few bugs to work out when I was rewriting the `link` code, but nothing compared to the mess I had to deal with when writing the old code (which was a couple of horrifying interdependent `for select` loops to build a state machine). + +So by now you're probably wondering what any of this looks like in practice. Just to give a generic example, suppose we have some struct with an exported function that needs to run code on a worker goroutine. We could end up with something like the following when writing Go in the CSP style: + +```Go + +// This is the function we want the worker to run. +func (n *NonActorStruct) theFunction(arg1 Type1, arg2 Type2) { + // this is where the code we actually care about goes, the rest is basically boilerplate +} + +// This is the struct that we want the worker to own and manipulate. +type NonActorStruct struct { + inputForTheFunction chan argsForTheFunction + // fields we care about, plus maybe more channels for other things +} + +// Needed to initialize the channel to a working state +func NewNonActorStruct() *NonActorStruct { + n := NonActorStruct{ + inputForTheFunction: make(chan argsForTheFunction), + } + return &n +} + +// This is just a helper struct to carry arguments for the function. +type argsForTheFunction struct { + Arg1 Type1 + Arg2 Type2 +} + +// This is the function we export. +func (n *NonActorStruct) RunTheFunction(arg1 Type1, arg2 Type2) { + n.inputForTheFunction<-argsForTheFunction{arg1, arg2} +} + +// This is needed to start the worker, otherwise things block. +func (n *NonActorStruct) Start() { + go func() { + for { + select{ + // cases for other things we may need to do would also be here + // presumably at least one is involved in safely shutting down + case args := <-n.inputForTheFunction: + // We could possibly have a switch statement here + // Then switch on the arg type to pick which function to run + n.theFunction(args.Arg1, args.Arg2) + } + } + }() +} + +// This is needed to stop the worker when we're done. +func (n *NonActorStruct) Stop() { + // Actual implemenation depends on what else the worker does in its loop, + // but it probably just sends a specific message and/or closes some channel. +} + +// Then to use the code, we have something like: +myStruct := NewNonActorStruct() +myStruct.Start() +defer myStruct.Stop() // Or arrange this to happen somewhere else +myStruct.RunTheFunction(arg1, arg2) +``` + + +When migrating to the actor model, the basic pattern that emerged was to embed a `phony.Inbox` into any `struct` we wanted to make into a `phony.Actor`, and then define functions of the struct like so: + +```Go + +// This is the function we want the worker to run. +func (a *ActorStruct) theFunction(arg1 Type1, arg2 Type2) { + // this is where the code we actually care about goes, the rest is basically boilerplate +} + +// This is the struct that we want the worker to own and manipulate. +type ActorStruct struct { + phony.Inbox // This defines the Act function, satisfying the Actor interface + // fields we care about +} + +// This is the function we export. +func (a *ActorStruct) RunTheFunction(from phony.Actor, arg1 Type1, arg2 Type2) { + a.Act(from, func() { + a.theFunction(arg1, arg2) + }) +} + +// And then to use it, an Actor x would run something like: +myActor := new(ActorStruct) +myActor.RunTheFunction(x, arg1, arg2) +``` + +And that's about it. The first argument to `myActor.RunTheFunction` also `nil`able, if we have non-`Actor` code that needs to send a message, it just means there's no backpressure to slow down the non-`Actor` code if it's sending messages faster than the `Actor` can handle them. A `phony.Block` function exists to help non-`Actor`s wait for an `Actor` to process a message before continuing, since this seems like a common enough use case (especially when a package wants to export a non-`Actor` interface that uses `Actor` code internally). + +What's great is that we don't need to think about starting or stopping workers, deadlocks and leaks are not possible outside of blocking operations (e.g. I/O), and we can add or reuse behaviors just as easily as any function. I find the code easier to read and reason about too. + +I/O is one rough spot, since an `Actor` can block on a `Read` or a `Write` and not process incoming messages as a result. This isn't really any worse than working with normal Go code, and the pattern we've adopted is to have separate `Actor`s for `Read` and `Write`, where one mostly just sits in a `Read` loop and sends the results (and/or error) somewhere whenever a `Read` finishes. These two workers can be children of some parent `Actor`, which is the only one the rest of the code needs to know about, and then all we need to remember to do is close the `ReadWriteCloser` (e.g. socket) at some point when we're done. This is the sort of thing that we'll eventually want to write a standard `struct` for, update our code everywhere to use it, and then never have to think about it again. In the mean time, we have a couple of very similar implementations for working with sockets or the tun/tap device. + +### Dénouement + +The Go language makes concurrency easy, but for some problems it can be difficult to do safely out-of-the-box. However, the language provides the tools needed to implement an actor model approach very easily. While I won't claim that the actor model is a panacea for all development woes, Yggdrasil by its very nature requires us to think about networks of nodes communicating asynchronously, so it makes sense to use a programming paradigm that lets us model that approach more explicitly in our code base. Outside of a couple of corner cases (namely blocking I/O for the network sockets and the tun/tap device), we expect this to obviate any need to even thing about deadlocks, make development easier moving forward, and generally lead to a better user experience as a result. The code migration is still a work in progress, but `Actor`s have replace `for select` workers along the hot paths through the code (minus 1 crypto worker pool in the session code) and will slowly replace synchronization primitives in the remaining code base. The current code has been merged into our `develop` branch, and I'm quite excited to see it land in Yggdrasil `v0.3.9`, along with the usual bug fixes and incremental improvements, which we plan to release in the near future. + From 90866ba39ddcc33e65ff18aaf5e7538715d47855 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sun, 1 Sep 2019 21:52:57 +0100 Subject: [PATCH 40/95] Update 2019-09-01-actors.md --- _posts/2019-09-01-actors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-09-01-actors.md b/_posts/2019-09-01-actors.md index abfae37..0b58810 100644 --- a/_posts/2019-09-01-actors.md +++ b/_posts/2019-09-01-actors.md @@ -33,7 +33,7 @@ When cycles of goroutines naively pass messages over channels, deadlocks are all Typically, when a goroutine is started, it continues to run until either the function returns or the program exits. For this reason, if a goroutine executes any statements which can block (such as a channel operation), it's important to include some `case` which signals that it's time to return. Forgetting to do this can result in goroutine leaks. [Never start a goroutine without knowing how it will stop](https://dave.cheney.net/2016/12/22/never-start-a-goroutine-without-knowing-how-it-will-stop), or so the experts say. -This is sometimes harder than it needs to be. To be blunt, the single producer N consumer cases are fine, you just close the channel and have all the consumers take this as a signal to exit. Anything involving multiple producers requires some sort of signaling to indicate that all producers have exited. Since you're using a channel already, the obvious option is a `select` statement with another channel that closes to signal shutdown, and then something like e.g. a `[sync.WaitGroup](https://golang.org/pkg/sync/#WaitGroup)` to wait for all producers to exit before closing the channel. Until your number of producers needs to change at runtime, and you realize that this races if you start to `Wait` before `Add`ing everything to the group, so you need to implement a custom counter, and be careful that additions and subtractions can also race and cause it to shut down early. And have fun solving it, because with how much `select` resists composition and code reuse, you're going to be implementing the same patterns over, and over, and over, and over... +This is sometimes harder than it needs to be. To be blunt, the single producer N consumer cases are fine, you just close the channel and have all the consumers take this as a signal to exit. Anything involving multiple producers requires some sort of signaling to indicate that all producers have exited. Since you're using a channel already, the obvious option is a `select` statement with another channel that closes to signal shutdown, and then something like e.g. a [`sync.WaitGroup`](https://golang.org/pkg/sync/#WaitGroup) to wait for all producers to exit before closing the channel. Until your number of producers needs to change at runtime, and you realize that this races if you start to `Wait` before `Add`ing everything to the group, so you need to implement a custom counter, and be careful that additions and subtractions can also race and cause it to shut down early. And have fun solving it, because with how much `select` resists composition and code reuse, you're going to be implementing the same patterns over, and over, and over, and over... It's not that this is some impossible problem to solve, it's just that Go's take on the [CSP](https://en.wikipedia.org/wiki/Communicating_sequential_processes), combined with the rest of the tools the language gives you, makes it easy and concise to run thing the *wrong* way, and leads to comparatively complex and delicate code when trying to run it the right way. At least, that's my personal view of it based on my experience so far, but it probably varies some based on the problem the code is trying to solve. From 8d76d73941c806b722a48522108ffab5a2714817 Mon Sep 17 00:00:00 2001 From: Arceliar Date: Sun, 1 Sep 2019 15:59:38 -0500 Subject: [PATCH 41/95] Update 2019-09-01-actors.md fix link to phony --- _posts/2019-09-01-actors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2019-09-01-actors.md b/_posts/2019-09-01-actors.md index 0b58810..d5b95d7 100644 --- a/_posts/2019-09-01-actors.md +++ b/_posts/2019-09-01-actors.md @@ -54,7 +54,7 @@ Different implementations differ on details after that, such as what order messa I'm particularly fond of the [pony](https://ponylang.io) programming language's take on the actor model. I really can't being to say enough nice things about their approach, and fully describing it is beyond the scope of this blog post, but if you come out of here with an interest in the actor model, then I highly recommend checking out that language. Maybe watch a few of the talks from the developers that have been posted to youtube, or read their papers about what is *easily* the most promising approach to garbage collection I've ever come across. -Anyway, I don't actually work on anything written in pony, but I like their version of the actor model so much that I decided to see if I could trick Go's runtime into faking it. The result is [`phony`](https://github.com/Arceliar/yggdrasil-go), which manages to do most of what I want in under 70 lines of code. When we write code using this asynchronous message passing style, instead of ordinary goroutines+channels, the implications are pretty significant: +Anyway, I don't actually work on anything written in pony, but I like their version of the actor model so much that I decided to see if I could trick Go's runtime into faking it. The result is [`phony`](https://github.com/Arceliar/phony), which manages to do most of what I want in under 70 lines of code. When we write code using this asynchronous message passing style, instead of ordinary goroutines+channels, the implications are pretty significant: 1. There are no deadlocks. Message sends always succeed, and are quite fast (it doesn't even require [CAS](https://en.wikipedia.org/wiki/Compare-and-swap) instructions in the normal case). 2. Inbox sizes stay small due to backpressure: if the sender sees that the receiver's inbox has too many pending messages, it will schedule itself to stop at some deadlock-free safe point in the future, to wait until the receiver signals that it's handled the message. From 2e5d4b34ab997bbf8005d99da017f1639c8e4591 Mon Sep 17 00:00:00 2001 From: passenger Date: Tue, 3 Sep 2019 20:17:46 +0200 Subject: [PATCH 42/95] add gemfile --- Gemfile | 9 ++ Gemfile.lock | 251 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 260 insertions(+) create mode 100644 Gemfile create mode 100644 Gemfile.lock diff --git a/Gemfile b/Gemfile new file mode 100644 index 0000000..f419341 --- /dev/null +++ b/Gemfile @@ -0,0 +1,9 @@ +gem "jekyll", "~> 3.8" + +gem "jekyll-theme-minimal", "~> 0.1.1" + +gem "jekyll-feed", "~> 0.11.0" + +gem "jekyll-sitemap", "~> 1.2" + +gem "github-pages", group: :jekyll_plugins diff --git a/Gemfile.lock b/Gemfile.lock new file mode 100644 index 0000000..71bab45 --- /dev/null +++ b/Gemfile.lock @@ -0,0 +1,251 @@ +GEM + specs: + activesupport (4.2.11.1) + i18n (~> 0.7) + minitest (~> 5.1) + thread_safe (~> 0.3, >= 0.3.4) + tzinfo (~> 1.1) + addressable (2.6.0) + public_suffix (>= 2.0.2, < 4.0) + coffee-script (2.4.1) + coffee-script-source + execjs + coffee-script-source (1.11.1) + colorator (1.1.0) + commonmarker (0.17.13) + ruby-enum (~> 0.5) + concurrent-ruby (1.1.5) + dnsruby (1.61.2) + addressable (~> 2.5) + em-websocket (0.5.1) + eventmachine (>= 0.12.9) + http_parser.rb (~> 0.6.0) + ethon (0.12.0) + ffi (>= 1.3.0) + eventmachine (1.2.7) + execjs (2.7.0) + faraday (0.15.4) + multipart-post (>= 1.2, < 3) + ffi (1.11.1) + forwardable-extended (2.6.0) + gemoji (3.0.1) + github-pages (198) + activesupport (= 4.2.11.1) + github-pages-health-check (= 1.16.1) + jekyll (= 3.8.5) + jekyll-avatar (= 0.6.0) + jekyll-coffeescript (= 1.1.1) + jekyll-commonmark-ghpages (= 0.1.5) + jekyll-default-layout (= 0.1.4) + jekyll-feed (= 0.11.0) + jekyll-gist (= 1.5.0) + jekyll-github-metadata (= 2.12.1) + jekyll-mentions (= 1.4.1) + jekyll-optional-front-matter (= 0.3.0) + jekyll-paginate (= 1.1.0) + jekyll-readme-index (= 0.2.0) + jekyll-redirect-from (= 0.14.0) + jekyll-relative-links (= 0.6.0) + jekyll-remote-theme (= 0.3.1) + jekyll-sass-converter (= 1.5.2) + jekyll-seo-tag (= 2.5.0) + jekyll-sitemap (= 1.2.0) + jekyll-swiss (= 0.4.0) + jekyll-theme-architect (= 0.1.1) + jekyll-theme-cayman (= 0.1.1) + jekyll-theme-dinky (= 0.1.1) + jekyll-theme-hacker (= 0.1.1) + jekyll-theme-leap-day (= 0.1.1) + jekyll-theme-merlot (= 0.1.1) + jekyll-theme-midnight (= 0.1.1) + jekyll-theme-minimal (= 0.1.1) + jekyll-theme-modernist (= 0.1.1) + jekyll-theme-primer (= 0.5.3) + jekyll-theme-slate (= 0.1.1) + jekyll-theme-tactile (= 0.1.1) + jekyll-theme-time-machine (= 0.1.1) + jekyll-titles-from-headings (= 0.5.1) + jemoji (= 0.10.2) + kramdown (= 1.17.0) + liquid (= 4.0.0) + listen (= 3.1.5) + mercenary (~> 0.3) + minima (= 2.5.0) + nokogiri (>= 1.8.5, < 2.0) + rouge (= 2.2.1) + terminal-table (~> 1.4) + github-pages-health-check (1.16.1) + addressable (~> 2.3) + dnsruby (~> 1.60) + octokit (~> 4.0) + public_suffix (~> 3.0) + typhoeus (~> 1.3) + html-pipeline (2.11.1) + activesupport (>= 2) + nokogiri (>= 1.4) + http_parser.rb (0.6.0) + i18n (0.9.5) + concurrent-ruby (~> 1.0) + jekyll (3.8.5) + addressable (~> 2.4) + colorator (~> 1.0) + em-websocket (~> 0.5) + i18n (~> 0.7) + jekyll-sass-converter (~> 1.0) + jekyll-watch (~> 2.0) + kramdown (~> 1.14) + liquid (~> 4.0) + mercenary (~> 0.3.3) + pathutil (~> 0.9) + rouge (>= 1.7, < 4) + safe_yaml (~> 1.0) + jekyll-avatar (0.6.0) + jekyll (~> 3.0) + jekyll-coffeescript (1.1.1) + coffee-script (~> 2.2) + coffee-script-source (~> 1.11.1) + jekyll-commonmark (1.3.1) + commonmarker (~> 0.14) + jekyll (>= 3.7, < 5.0) + jekyll-commonmark-ghpages (0.1.5) + commonmarker (~> 0.17.6) + jekyll-commonmark (~> 1) + rouge (~> 2) + jekyll-default-layout (0.1.4) + jekyll (~> 3.0) + jekyll-feed (0.11.0) + jekyll (~> 3.3) + jekyll-gist (1.5.0) + octokit (~> 4.2) + jekyll-github-metadata (2.12.1) + jekyll (~> 3.4) + octokit (~> 4.0, != 4.4.0) + jekyll-mentions (1.4.1) + html-pipeline (~> 2.3) + jekyll (~> 3.0) + jekyll-optional-front-matter (0.3.0) + jekyll (~> 3.0) + jekyll-paginate (1.1.0) + jekyll-readme-index (0.2.0) + jekyll (~> 3.0) + jekyll-redirect-from (0.14.0) + jekyll (~> 3.3) + jekyll-relative-links (0.6.0) + jekyll (~> 3.3) + jekyll-remote-theme (0.3.1) + jekyll (~> 3.5) + rubyzip (>= 1.2.1, < 3.0) + jekyll-sass-converter (1.5.2) + sass (~> 3.4) + jekyll-seo-tag (2.5.0) + jekyll (~> 3.3) + jekyll-sitemap (1.2.0) + jekyll (~> 3.3) + jekyll-swiss (0.4.0) + jekyll-theme-architect (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-cayman (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-dinky (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-hacker (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-leap-day (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-merlot (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-midnight (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-minimal (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-modernist (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-primer (0.5.3) + jekyll (~> 3.5) + jekyll-github-metadata (~> 2.9) + jekyll-seo-tag (~> 2.0) + jekyll-theme-slate (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-tactile (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-time-machine (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-titles-from-headings (0.5.1) + jekyll (~> 3.3) + jekyll-watch (2.2.1) + listen (~> 3.0) + jemoji (0.10.2) + gemoji (~> 3.0) + html-pipeline (~> 2.2) + jekyll (~> 3.0) + kramdown (1.17.0) + liquid (4.0.0) + listen (3.1.5) + rb-fsevent (~> 0.9, >= 0.9.4) + rb-inotify (~> 0.9, >= 0.9.7) + ruby_dep (~> 1.2) + mercenary (0.3.6) + mini_portile2 (2.4.0) + minima (2.5.0) + jekyll (~> 3.5) + jekyll-feed (~> 0.9) + jekyll-seo-tag (~> 2.1) + minitest (5.11.3) + multipart-post (2.1.1) + nokogiri (1.10.3) + mini_portile2 (~> 2.4.0) + octokit (4.14.0) + sawyer (~> 0.8.0, >= 0.5.3) + pathutil (0.16.2) + forwardable-extended (~> 2.6) + public_suffix (3.1.1) + rb-fsevent (0.10.3) + rb-inotify (0.10.0) + ffi (~> 1.0) + rouge (2.2.1) + ruby-enum (0.7.2) + i18n + ruby_dep (1.5.0) + rubyzip (1.2.3) + safe_yaml (1.0.5) + sass (3.7.4) + sass-listen (~> 4.0.0) + sass-listen (4.0.0) + rb-fsevent (~> 0.9, >= 0.9.4) + rb-inotify (~> 0.9, >= 0.9.7) + sawyer (0.8.2) + addressable (>= 2.3.5) + faraday (> 0.8, < 2.0) + terminal-table (1.8.0) + unicode-display_width (~> 1.1, >= 1.1.1) + thread_safe (0.3.6) + typhoeus (1.3.1) + ethon (>= 0.9.0) + tzinfo (1.2.5) + thread_safe (~> 0.1) + unicode-display_width (1.6.0) + +PLATFORMS + ruby + +DEPENDENCIES + github-pages + jekyll (~> 3.8) + jekyll-feed (~> 0.11.0) + jekyll-sitemap (~> 1.2) + jekyll-theme-minimal (~> 0.1.1) + +BUNDLED WITH + 1.16.1 From 000b6aae6e50c4a5c25aa6342b7d915d8098e8f6 Mon Sep 17 00:00:00 2001 From: passenger Date: Tue, 3 Sep 2019 20:20:29 +0200 Subject: [PATCH 43/95] fix mobile overflow --- assets/css/style.scss | 1 + 1 file changed, 1 insertion(+) diff --git a/assets/css/style.scss b/assets/css/style.scss index 43cf8e1..830cedc 100644 --- a/assets/css/style.scss +++ b/assets/css/style.scss @@ -30,6 +30,7 @@ section { position: relative; flex: initial; word-break: break-word; + overflow: hidden; table { thead { th { From b96573a3c6bce0c2228c151f7075b00b55691745 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 4 Sep 2019 19:41:51 +0100 Subject: [PATCH 44/95] Removing Umbrellix services for now as they seem to be unreachable cc @janicez - let me know if you fix these and I'll revert this commit! --- services.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/services.md b/services.md index 657fe95..a100045 100644 --- a/services.md +++ b/services.md @@ -59,13 +59,6 @@ The following services are available on the Yggdrasil network, courtesy of our c - cathugger's node via Yggdrasil at `201:4806:21d5:c971:407f:4ea9:4d7d:e491` port `6667` (TCP) - Jeff's node via Yggdrasil at `203:e0b0:ec08:4e1f:f004:19a9:577a:90ba` port `6667` (TCP) -### Umbrellix - -- Main channel is `#lounge` -- Accessible via Yggdrasil at `202:8fb5:1490:594d:7e29:98fd:8d79:953f` ports `194` (TCP), `994` (TCP+SSL), `6667` (TCP), `6697` (TCP+SSL) -- Accessible via cjdns at `fcda:c930:1c80:ef8a:4fb:6c65:8506:1dd2` ports `194` (TCP), `994` (TCP+SSL), `6667` (TCP), `6697` (TCP+SSL) -- It's preferred that you act like you're just another random clearnetter - Umbrellix is primarily a clearnet IRC network - ### PirateIRC - `y.irc.pirateirc.net` accessible via Yggdrasil at `201:677d:1809:c5eb:202a:d39:e598:305` port `6697` (TCP+SSL) @@ -170,15 +163,6 @@ The following services are available on the Yggdrasil network, courtesy of our c ---- -## Gopher Holes - -### Umbrellix Gopher - - - `gopher-y.umbrellix.net` or `202:8fb5:1490:594d:7e29:98fd:8d79:953f` - - Hole not updated with details of Yggdrasil, yet. - ----- - ## Direct Connect - bepis DC-hub run by *Jeff* `adc://[203:e0b0:ec08:4e1f:f004:19a9:577a:90ba]:1511` From 6a37efe6577835534d889b09df0b0f8717b395a6 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 4 Sep 2019 22:02:27 +0100 Subject: [PATCH 45/95] Revert "Removing Umbrellix services for now as they seem to be unreachable" This reverts commit b96573a3c6bce0c2228c151f7075b00b55691745. --- services.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/services.md b/services.md index a100045..657fe95 100644 --- a/services.md +++ b/services.md @@ -59,6 +59,13 @@ The following services are available on the Yggdrasil network, courtesy of our c - cathugger's node via Yggdrasil at `201:4806:21d5:c971:407f:4ea9:4d7d:e491` port `6667` (TCP) - Jeff's node via Yggdrasil at `203:e0b0:ec08:4e1f:f004:19a9:577a:90ba` port `6667` (TCP) +### Umbrellix + +- Main channel is `#lounge` +- Accessible via Yggdrasil at `202:8fb5:1490:594d:7e29:98fd:8d79:953f` ports `194` (TCP), `994` (TCP+SSL), `6667` (TCP), `6697` (TCP+SSL) +- Accessible via cjdns at `fcda:c930:1c80:ef8a:4fb:6c65:8506:1dd2` ports `194` (TCP), `994` (TCP+SSL), `6667` (TCP), `6697` (TCP+SSL) +- It's preferred that you act like you're just another random clearnetter - Umbrellix is primarily a clearnet IRC network + ### PirateIRC - `y.irc.pirateirc.net` accessible via Yggdrasil at `201:677d:1809:c5eb:202a:d39:e598:305` port `6697` (TCP+SSL) @@ -163,6 +170,15 @@ The following services are available on the Yggdrasil network, courtesy of our c ---- +## Gopher Holes + +### Umbrellix Gopher + + - `gopher-y.umbrellix.net` or `202:8fb5:1490:594d:7e29:98fd:8d79:953f` + - Hole not updated with details of Yggdrasil, yet. + +---- + ## Direct Connect - bepis DC-hub run by *Jeff* `adc://[203:e0b0:ec08:4e1f:f004:19a9:577a:90ba]:1511` From 993bc6db500251d5d3933d534f844cf0192ca5dc Mon Sep 17 00:00:00 2001 From: "Stanislav N. aka pztrn" Date: Fri, 6 Sep 2019 14:20:22 +0500 Subject: [PATCH 46/95] Added information about pztrn's NNTP server and removed i2p URLs for Meshforum as they're not working ATM. --- services.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/services.md b/services.md index 657fe95..76127a2 100644 --- a/services.md +++ b/services.md @@ -26,7 +26,6 @@ The following services are available on the Yggdrasil network, courtesy of our c - [Mesh Forum](http://[303:60d4:3d32:a2b9::4]/) - a chit-chat forum about everything. Run by @pztrn. - Accessible via Yggdrasil at `http://[303:60d4:3d32:a2b9::4]/`. - - Accessible via I2P at `http://meshforum.i2p` or `http://hp2uzysaur74tbeojtlcsl7apsy2jgqxydd7b5asgvztihz76jia.b32.i2p/`. ### Wikis @@ -111,6 +110,20 @@ The following services are available on the Yggdrasil network, courtesy of our c ---- +## NNTP + +### pztrn's NNTP server + +- [Website with rules, news and documentation](https://nntp.pztrn.name). +- Registration required to access all groups beyond `local.regreq` which is used for registration requests. No anonymous postings allowed. +- ygg.* hierarchy for talks about Yggdrasil! :) +- Accessible via Yggdrasil at `201:e64d:1a9:d2b0:9c74:f285:ea73:5e05` ports `119` (plain text) and `563` (TLSv1.0+ only). +- Accessible via public Internet at `nntp.pztrn.name` ports `119` (plain text) and `563` (TLSv1.0+ only). +- SSL warnings via Yggdrasil expected, certificate is for *.pztrn.name. You can workaround it by adding entry in hosts file. +- Peering is welcome, especially via Yggdrasil. + +---- + ## BitTorrent - [neilalexander](https://matrix.to/#/@neilalexander:matrix.org)'s public tracker available at `http://tracker.y.neilalexander.eu:6969/announce` or `http://[301:4541:2f84:1188:216:3eff:fe34:ec44]:6969/announce` From 32fd0493c292b53ca2993e8f8db7885d2df8f5bf Mon Sep 17 00:00:00 2001 From: passenger Date: Sat, 7 Sep 2019 12:26:10 +0200 Subject: [PATCH 47/95] fix height for burger menu --- assets/css/style.scss | 1 - 1 file changed, 1 deletion(-) diff --git a/assets/css/style.scss b/assets/css/style.scss index 830cedc..e64e727 100644 --- a/assets/css/style.scss +++ b/assets/css/style.scss @@ -81,7 +81,6 @@ header { position: initial; } position: fixed; - height: 100%; overflow-y: scroll; background-color: white; top: 0px; From 68fd565b468c0920db44ab39684f9d27db63607c Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 11 Sep 2019 20:33:37 +0100 Subject: [PATCH 48/95] fix listen --- configuration.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/configuration.md b/configuration.md index 9b864d8..dcc6029 100644 --- a/configuration.md +++ b/configuration.md @@ -43,7 +43,9 @@ A new configuration file has the following format. Please note that some of the { # Listen address for peer connections. Default is to listen for all # TCP connections over IPv4 and IPv6 with a random port. - Listen: "[::]:xxxxx" + Listen: [ + tcp://[::]:xxxxx + ] # Listen address for admin connections Default is to listen for local # connections either on TCP/9001 or a UNIX socket depending on your @@ -186,9 +188,8 @@ Note that any field not specified in the configuration will use its default valu ## Configuration Options - `Listen` - - A string, in the form of `"ip:port"`, on which to listen for (TCP) connections from peers. + - A list of strings in the form `[ "tcp://listenAddress:listenPort", ... ]`, on which to listen for (TCP) connections from peers. - Note that, due to Go language design choices, `[::]` listens on IPv4 and IPv6 on most platforms, while an empty IP or `0.0.0.0` listens only to IPv4. - - The default is to listen on all addresses (`[::]`) with a random port. - `AdminListen` - Port to listen on for the admin socket, specified in URI format, i.e. `tcp://localhost:9001`. - On supported platforms, the admin socket can listen on a UNIX domain socket instead, i.e. `unix:///var/run/yggdrasil.sock`. From 8ba2b4aac70a7e61013978b3486cfaf64c82e1ba Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 11 Sep 2019 20:35:32 +0100 Subject: [PATCH 49/95] Add examples for peers/interfacepeers --- configuration.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/configuration.md b/configuration.md index dcc6029..a6f28b5 100644 --- a/configuration.md +++ b/configuration.md @@ -54,13 +54,21 @@ A new configuration file has the following format. Please note that some of the # List of connection strings for static peers in URI format, e.g. # tcp://a.b.c.d:e or socks://a.b.c.d:e/f.g.h.i:j. - Peers: [] + Peers: [ + tcp://a.b.c.d:xxxxx + tcp://d.c.b.a:xxxxx + ] # List of connection strings for static peers in URI format, arranged # by source interface, e.g. { "eth0": [ tcp://a.b.c.d:e ] }. Note that # SOCKS peerings will NOT be affected by this option and should go in # the "Peers" section instead. - InterfacePeers: {} + InterfacePeers: { + "eth0": [ + tcp://a.b.c.d:xxxxx + tcp://d.c.b.a:xxxxx + ] + } # Read timeout for connections, specified in milliseconds. If less # than 6000 and not negative, 6000 (the default) is used. If negative, From 7a2b32972d390fee9a3f9660a4e2f1dd64f1bc11 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 11 Sep 2019 20:38:28 +0100 Subject: [PATCH 50/95] Fix configuration sample --- configuration.md | 106 +++++++++++++++++++++++++++-------------------- 1 file changed, 60 insertions(+), 46 deletions(-) diff --git a/configuration.md b/configuration.md index a6f28b5..39ad0db 100644 --- a/configuration.md +++ b/configuration.md @@ -41,28 +41,19 @@ A new configuration file has the following format. Please note that some of the ``` { - # Listen address for peer connections. Default is to listen for all - # TCP connections over IPv4 and IPv6 with a random port. - Listen: [ - tcp://[::]:xxxxx - ] - - # Listen address for admin connections Default is to listen for local - # connections either on TCP/9001 or a UNIX socket depending on your - # platform. Use this value for yggdrasilctl -endpoint=X. - AdminListen: tcp://localhost:9001 - - # List of connection strings for static peers in URI format, e.g. - # tcp://a.b.c.d:e or socks://a.b.c.d:e/f.g.h.i:j. + # List of connection strings for outbound peer connections in URI format, + # e.g. tcp://a.b.c.d:e or socks://a.b.c.d:e/f.g.h.i:j. These connections + # will obey the operating system routing table, therefore you should + # use this section when you may connect via different interfaces. Peers: [ tcp://a.b.c.d:xxxxx tcp://d.c.b.a:xxxxx ] - # List of connection strings for static peers in URI format, arranged - # by source interface, e.g. { "eth0": [ tcp://a.b.c.d:e ] }. Note that - # SOCKS peerings will NOT be affected by this option and should go in - # the "Peers" section instead. + # List of connection strings for outbound peer connections in URI format, + # arranged by source interface, e.g. { "eth0": [ tcp://a.b.c.d:e ] }. + # Note that SOCKS peerings will NOT be affected by this option and should + # go in the "Peers" section instead. InterfacePeers: { "eth0": [ tcp://a.b.c.d:xxxxx @@ -70,14 +61,33 @@ A new configuration file has the following format. Please note that some of the ] } - # Read timeout for connections, specified in milliseconds. If less - # than 6000 and not negative, 6000 (the default) is used. If negative, - # reads won't time out. - ReadTimeout: 0 + # Listen addresses for incoming connections. You will need to add + # listeners in order to accept incoming peerings from non-local nodes. + # Multicast peer discovery will work regardless of any listeners set + # here. Each listener should be specified in URI format as above, e.g. + # tcp://0.0.0.0:0 or tcp://[::]:0 to listen on all interfaces. + Listen: [ + tcp://[::]:xxxxx + ] - # List of peer encryption public keys to allow or incoming TCP - # connections from. If left empty/undefined then all connections - # will be allowed by default. + # Listen address for admin connections. Default is to listen for local + # connections either on TCP/9001 or a UNIX socket depending on your + # platform. Use this value for yggdrasilctl -endpoint=X. To disable + # the admin socket, use the value "none" instead. + AdminListen: tcp://localhost:9001 + + # Regular expressions for which interfaces multicast peer discovery + # should be enabled on. If none specified, multicast peer discovery is + # disabled. The default value is .* which uses all interfaces. + MulticastInterfaces: + [ + .* + ] + + # List of peer encryption public keys to allow incoming TCP peering + # connections from. If left empty/undefined then all connections will + # be allowed by default. This does not affect outgoing peerings, nor + # does it affect link-local peers discovered via multicast. AllowedEncryptionPublicKeys: [] # Your public encryption key. Your peers may ask you for this to put @@ -94,13 +104,12 @@ A new configuration file has the following format. Please note that some of the # Your private signing key. DO NOT share this with anyone! SigningPrivateKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx - # Regular expressions for which interfaces multicast peer discovery - # should be enabled on. If none specified, multicast peer discovery is - # disabled. The default value is .* which uses all interfaces. - MulticastInterfaces: - [ - .* - ] + # The port number to be used for the link-local TCP listeners for the + # configured MulticastInterfaces. This option does not affect listeners + # specified in the Listen option. Unless you plan to firewall link-local + # traffic, it is best to leave this as the default value of 0. This + # option cannot currently be changed by reloading config during runtime. + LinkLocalTCPPort: 0 # Local network interface name for TUN/TAP adapter, or "auto" to select # an interface automatically, or "none" to run without TUN/TAP. @@ -156,24 +165,22 @@ A new configuration file has the following format. Please note that some of the # Enable or disable tunnel routing. Enable: false - # IPv6 CIDR subnets, mapped to the EncryptionPublicKey to which they - # should be routed, e.g. { "aaaa:bbbb:cccc::/e": "boxpubkey", ... } - IPv6Destinations: {} + # IPv6 subnets belonging to remote nodes, mapped to the node's public + # key, e.g. { "aaaa:bbbb:cccc::/e": "boxpubkey", ... } + IPv6RemoteSubnets: {} - # Optional IPv6 source subnets which are allowed to be tunnelled in - # addition to this node's Yggdrasil address/subnet. If not - # specified, only traffic originating from this node's Yggdrasil - # address or subnet will be tunnelled. - IPv6Sources: [] + # IPv6 subnets belonging to this node's end of the tunnels. Only traffic + # from these ranges (or the Yggdrasil node's IPv6 address/subnet) + # will be tunnelled. + IPv6LocalSubnets: [] - # IPv4 CIDR subnets, mapped to the EncryptionPublicKey to which they - # should be routed, e.g. { "a.b.c.d/e": "boxpubkey", ... } - IPv4Destinations: {} + # IPv4 subnets belonging to remote nodes, mapped to the node's public + # key, e.g. { "a.b.c.d/e": "boxpubkey", ... } + IPv4RemoteSubnets: {} - # IPv4 source subnets which are allowed to be tunnelled. Unlike for - # IPv6, this option is required for bridging IPv4 traffic. Only - # traffic with a source matching these subnets will be tunnelled. - IPv4Sources: [] + # IPv4 subnets belonging to this node's end of the tunnels. Only traffic + # from these ranges will be tunnelled. + IPv4LocalSubnets: [] } # Advanced options for tuning the switch. Normally you will not need @@ -184,6 +191,13 @@ A new configuration file has the following format. Please note that some of the MaxTotalQueueSize: 4194304 } + # By default, nodeinfo contains some defaults including the platform, + # architecture and Yggdrasil version. These can help when surveying + # the network and diagnosing network routing problems. Enabling + # nodeinfo privacy prevents this, so that only items specified in + # "NodeInfo" are sent back if specified. + NodeInfoPrivacy: false + # Optional node info. This must be a { "key": "value", ... } map # or set as null. This is entirely optional but, if set, is visible # to the whole network on request. From c876890a51d9140e68d5cec7fbeb2146c2562792 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 27 Sep 2019 09:48:59 +0100 Subject: [PATCH 51/95] Update changelog.md --- changelog.md | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/changelog.md b/changelog.md index b0be64d..fdac254 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,34 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> +## [0.3.9] - 2019-09-27 +### Added +- Yggdrasil will now complain more verbosely when a peer URI is incorrectly formatted +- Soft-shutdown methods have been added, allowing a node to shut down gracefully when terminated +- New multicast interval logic which sends multicast beacons more often when Yggdrasil is first started to increase the chance of finding nearby nodes quickly after startup + +### Changed +- The switch now buffers packets more eagerly in an attempt to give the best link a chance to send, which appears to reduce packet reordering when crossing aggregate sets of peerings +- Substantial amounts of the codebase have been refactored to use the actor model, which should substantially reduce the chance of deadlocks +- Nonce tracking in sessions has been modified so that memory usage is reduced whilst still only allowing duplicate packets within a small window +- Soft-reconfiguration support has been simplified using new actor functions +- The garbage collector threshold has been adjusted for mobile builds +- The maximum queue size is now managed exclusively by the switch rather than by the core + +### Fixed +- The broken `hjson-go` dependency which affected builds of the previous version has now been resolved in the module manifest +- Some minor memory leaks in the switch have been fixed, which improves memory usage on mobile builds +- A memory leak in the add-peer loop has been fixed +- The admin socket now reports the correct URI strings for SOCKS peers in `getPeers` +- A race condition when dialling a remote node by both the node address and routed prefix simultaneously has been fixed +- A race condition between the router and the dial code resulting in a panic has been fixed +- A panic which could occur when the TUN/TAP interface disappears (e.g. during soft-shutdown) has been fixed +- A bug in the semantic versioning script which accompanies Yggdrasil for builds has been fixed +- A panic which could occur when the TUN/TAP interface reads an undersized/corrupted packet has been fixed + +### Removed +- A number of legacy debug functions have now been removed and a number of exported API functions are now better documented + ## [0.3.8] - 2019-08-21 ### Changed - Yggdrasil can now send multiple packets from the switch at once, which results in improved throughput with smaller packets or lower MTUs @@ -39,12 +67,14 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - New nonce tracking should help to reduce the number of packets dropped as a result of multiple/aggregate paths or congestion control in the switch ### Fixed -- **Security vulnerability**: Address verification was not strict enough, which could result in a malicious session sending traffic with unexpected or spoofed source or destination addresses which Yggdrasil could fail to reject - - Versions `0.3.6` and `0.3.7` are vulnerable - users of these versions should upgrade as soon as possible - - Versions `0.3.5` and earlier are not affected - A deadlock was fixed in the session code which could result in Yggdrasil failing to pass traffic after some time -## [0.3.7] - 2019-08-18 +### Security +- Address verification was not strict enough, which could result in a malicious session sending traffic with unexpected or spoofed source or destination addresses which Yggdrasil could fail to reject + - Versions `0.3.6` and `0.3.7` are vulnerable - users of these versions should upgrade as soon as possible + - Versions `0.3.5` and earlier are not affected + +## [0.3.7] - 2019-08-14 ### Changed - The switch should now forward packets along a single path more consistently in cases where congestion is low and multiple equal-length paths exist, which should improve stability and result in fewer out-of-order packets - Sessions should now be more tolerant of out-of-order packets, by replacing a bitmask with a variable sized heap+map structure to track recently received nonces, which should reduce the number of packets dropped due to reordering when multiple paths are used or multiple independent flows are transmitted through the same session From d3aebbccccb8b6ef4e762b0ed80497d0f57f44cb Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 27 Sep 2019 10:00:38 +0100 Subject: [PATCH 52/95] Update index.md --- index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.md b/index.md index ed5d292..73ff4e1 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.8](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.9](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. From ba1cbb9ee6b3b2e90c0d26766b20e6fc850e30ee Mon Sep 17 00:00:00 2001 From: abslimit Date: Sat, 5 Oct 2019 19:31:42 +0300 Subject: [PATCH 53/95] Deleted matrix and added XMPP and xonotic Deleted matrix server and added XMPP server. Matrix server is not running anymore. XMPP server is prioritized for yggdrasil. Added Xonotic multiplayer FPS for testing real-time apps. --- services.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/services.md b/services.md index 76127a2..5bcbd22 100644 --- a/services.md +++ b/services.md @@ -84,11 +84,14 @@ The following services are available on the Yggdrasil network, courtesy of our c ---- -## Matrix +## XMPP ### netwhood.online -- `http://[202:12a9:e5:4474:d473:82be:16ac:9381]:8008` run by [abslimit](http://netwhood.online/feedback/) +- `netwhood.online`, run by [abslimit](http://netwhood.online/feedback/) + - SRV record `_xmpp-client._tcp` is prioritized for `y.netwhood.online` (`202:12a9:e5:4474:d473:82be:16ac:9381`). XMPP-client must try yggdrasil address at first. + - SRV record `_xmpp-server._tcp` is prioritized for `y.netwhood.online` too. XMPP s2s connections must try yggdrasil address at first. Lets peer? + - "HTTP upload" XEP module is tied to `netwhood.online`, it will connect through the Internet. It can not be prioritized for yggdrasil. ---- @@ -146,6 +149,10 @@ The following services are available on the Yggdrasil network, courtesy of our c ### Minetest - `300:aa35:f9c1:dcce::3:30000`, admin Admin + +### Xonotic + - `201:9d54:3c57:d6d2:e8d7:a8ce:841f:eb89:26000` + - DNS name: `xonotic.y.netwhood.online:26000` ---- From 3d802a6eacaa8ff922d2b9a43c6be9a284eca6a1 Mon Sep 17 00:00:00 2001 From: Arceliar Date: Sat, 5 Oct 2019 12:39:12 -0500 Subject: [PATCH 54/95] Update platform-linux.md --- platform-linux.md | 1 + 1 file changed, 1 insertion(+) diff --git a/platform-linux.md b/platform-linux.md index bec7cb1..e01a3ab 100644 --- a/platform-linux.md +++ b/platform-linux.md @@ -13,3 +13,4 @@ Yggdrasil is well supported on Linux. - The maximum MTU size supported on Linux is 65535 in TUN mode and 65521 in TAP mode. - IPv6 needs to be enabled in order for Yggdrasil to work - IPv6 is usually enabled by default, but if not, enable using `sysctl -w net.ipv6.conf.all.disable_ipv6=0` or similar. - If using TUN/TAP then `/dev/net/tun` should be present on your system. +- The [BBR](https://github.com/google/bbr) congestion control algorithm is strongly recommended, as it can significantly reduce latency when under load. On recent distributions, this can be done with `sysctl -w net.ipv4.tcp_congestion_control=bbr` or similar. From 34313a621d31185240bc13b64972e1a460f42f10 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 10 Oct 2019 21:21:44 +0100 Subject: [PATCH 55/95] Update changelog.md --- changelog.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/changelog.md b/changelog.md index fdac254..bbf0bfa 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,17 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> +## [0.3.10] - 2019-10-10 +### Added +- The core library now includes several unit tests for peering and `yggdrasil.Conn` connections + +### Changed +- On recent Linux kernels, Yggdrasil will now set the `tcp_congestion_control` algorithm used for its own TCP sockets to [BBR](https://github.com/google/bbr), which reduces latency under load +- The systemd service configuration in `contrib` (and, by extension, some of our packages) now attemps to load the `tun` module, in case TUN/TAP support is available but not loaded, and it restricts Yggdrasil to the `CAP_NET_ADMIN` capability for managing the TUN/TAP adapter, rather than letting it do whatever the (typically `root`) user can do + +### Fixed +- The `yggdrasil.Conn.RemoteAddr()` function no longer blocks, fixing a deadlock when CKR is used while under heavy load + ## [0.3.9] - 2019-09-27 ### Added - Yggdrasil will now complain more verbosely when a peer URI is incorrectly formatted From 87cb8379459137a05bf4fa4801ef76d410cf9f43 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Thu, 10 Oct 2019 21:22:00 +0100 Subject: [PATCH 56/95] Update index.md --- index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.md b/index.md index 73ff4e1..c9720b4 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.9](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.10](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. From c27f9df654dfdcde7ff3dc05a2ce08bd58862933 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 25 Oct 2019 09:45:30 +0100 Subject: [PATCH 57/95] Update changelog.md --- changelog.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/changelog.md b/changelog.md index bbf0bfa..903ffc7 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,24 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> + +## [0.3.11] - 2019-10-25 +### Added +- Support for TLS listeners and peers has been added, allowing the use of `tls://host:port` in `Peers`, `InterfacePeers` and `Listen` configuration settings - this allows hiding Yggdrasil peerings inside regular TLS connections + +### Changed +- Go 1.13 or later is now required for building Yggdrasil +- Some exported API functions have been updated to work with standard Go interfaces: + - `net.Conn` instead of `yggdrasil.Conn` + - `net.Dialer` (the interface it would satisfy if it wasn't a concrete type) instead of `yggdrasil.Dialer` + - `net.Listener` instead of `yggdrasil.Listener` +- Session metadata is now updated correctly when a search completes for a node to which we already have an open session +- Multicast module reloading behaviour has been improved + +### Fixed +- An incorrectly held mutex in the crypto-key routing code has been fixed +- Multicast module no longer opens a listener socket if no multicast interfaces are configured + ## [0.3.10] - 2019-10-10 ### Added - The core library now includes several unit tests for peering and `yggdrasil.Conn` connections From ba7907459a2a92f2179169529464ba44b6a5fd4c Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 25 Oct 2019 09:45:45 +0100 Subject: [PATCH 58/95] Update index.md --- index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.md b/index.md index c9720b4..2f48d77 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.10](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.11](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. From 46fd84f90ee26564000bd788335fdb531e12e586 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 25 Oct 2019 10:35:42 +0100 Subject: [PATCH 59/95] Update services.md --- services.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/services.md b/services.md index 22013e2..b837fd7 100644 --- a/services.md +++ b/services.md @@ -173,7 +173,7 @@ The following services are available on the Yggdrasil network, courtesy of our c - `200:d0c4:68ee:e87b:c206:67b8:5fa5:d4be` port 53, hosted by [Medium](https://github.com/medium-isp) - Internal Yggdrasil services resolver - `200:5bf2:a7a5:27c7:54b8:6669:eb74:1813` port 53, hosted by [User2k](https://user2k.eu) - - Medium DNS + OpenDNS + Google DNS + CloudFlare DNS + - Medium DNS, OpenDNS, Google DNS and CloudFlare DNS ---- ## ZeroNet From d2f8dafa2f39580b52a3b40146ba70dcf364707b Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 25 Oct 2019 13:20:21 +0100 Subject: [PATCH 60/95] Update configuration.md --- configuration.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/configuration.md b/configuration.md index 39ad0db..d8296bc 100644 --- a/configuration.md +++ b/configuration.md @@ -47,7 +47,8 @@ A new configuration file has the following format. Please note that some of the # use this section when you may connect via different interfaces. Peers: [ tcp://a.b.c.d:xxxxx - tcp://d.c.b.a:xxxxx + socks://e.f.g.h:xxxxx/a.b.c.d:xxxxx + tls://a.b.c.d:xxxxx ] # List of connection strings for outbound peer connections in URI format, @@ -57,7 +58,7 @@ A new configuration file has the following format. Please note that some of the InterfacePeers: { "eth0": [ tcp://a.b.c.d:xxxxx - tcp://d.c.b.a:xxxxx + tls://a.b.c.d:xxxxx ] } @@ -68,6 +69,7 @@ A new configuration file has the following format. Please note that some of the # tcp://0.0.0.0:0 or tcp://[::]:0 to listen on all interfaces. Listen: [ tcp://[::]:xxxxx + tls://[::]:xxxxx ] # Listen address for admin connections. Default is to listen for local @@ -210,19 +212,21 @@ Note that any field not specified in the configuration will use its default valu ## Configuration Options - `Listen` - - A list of strings in the form `[ "tcp://listenAddress:listenPort", ... ]`, on which to listen for (TCP) connections from peers. + - A list of strings in the form `[ "tcp://listenAddress:listenPort", "tls://listenAddress:listenPort", ... ]`, on which to listen for TCP or TLS connections from peers. - Note that, due to Go language design choices, `[::]` listens on IPv4 and IPv6 on most platforms, while an empty IP or `0.0.0.0` listens only to IPv4. + - A `tcp://` listener can only accept `tcp://` peer connections, and a `tls://` listener can only accept `tls://` peer connections - `AdminListen` - Port to listen on for the admin socket, specified in URI format, i.e. `tcp://localhost:9001`. - On supported platforms, the admin socket can listen on a UNIX domain socket instead, i.e. `unix:///var/run/yggdrasil.sock`. - The default is to listen on the loopback interface (`tcp://localhost:9001`) which ensures that only local connections to the admin socket are allowed. - Note that if you change the listen address to a non-loopback address, this may allow other hosts on the network to manage the Yggdrasil process. This probably isn't desirable. - `Peers` - - A list of strings in the form `[ "tcp://peerAddress:peerPort", "socks://proxyAddress:proxyPort/peerAddress:peerPort", ... ]` of peers to connect to. + - A list of strings in the form `[ "tcp://peerAddress:peerPort", "tls://peerAddress:peerPort", "socks://proxyAddress:proxyPort/peerAddress:peerPort", ... ]` of peers to connect to. - Peer hostnames can be specified either using IPv4 addresses, IPv6 addresses or DNS names. - Each entry should begin with `tcp://` or `socks://proxyAddress:proxyPort/`. - `InterfacePeers` - - Like peers above, but arranged using specific interface names: `{ "eth0": [ "tcp://peerAddress:peerPort", "socks://proxyAddress:proxyPort/peerAddress:peerPort", ... ], "eth1": [], ... }` of peers to connect to. + - Like peers above, but arranged using specific interface names: `{ "eth0": [ "tcp://peerAddress:peerPort", "tls://peerAddress:peerPort", "socks://proxyAddress:proxyPort/peerAddress:peerPort", ... ], "eth1": [], ... }` of peers to connect to. + - A `tcp://` peer connection can only connect to a `tcp://` listener, and a `tls://` peer connection can only connect to a `tls://` listener - `AllowedEncryptionPublicKeys` - A list of strings in the form `["key", "key", ...]`, where `key` is each node's `EncryptionPublicKey` key which you would like to allow connections from. - This option allows you to restrict which other nodes can connect to your Yggdrasil node as a peer. It applies to incoming TCP connections. From c951b388ce290ea9ef051999ef0fab8459026f6b Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 25 Oct 2019 13:21:05 +0100 Subject: [PATCH 61/95] Update configuration.md --- configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/configuration.md b/configuration.md index d8296bc..41fbc59 100644 --- a/configuration.md +++ b/configuration.md @@ -214,7 +214,7 @@ Note that any field not specified in the configuration will use its default valu - `Listen` - A list of strings in the form `[ "tcp://listenAddress:listenPort", "tls://listenAddress:listenPort", ... ]`, on which to listen for TCP or TLS connections from peers. - Note that, due to Go language design choices, `[::]` listens on IPv4 and IPv6 on most platforms, while an empty IP or `0.0.0.0` listens only to IPv4. - - A `tcp://` listener can only accept `tcp://` peer connections, and a `tls://` listener can only accept `tls://` peer connections + - Note that a `tcp://` listener can only accept `tcp://` peer connections, and a `tls://` listener can only accept `tls://` peer connections - `AdminListen` - Port to listen on for the admin socket, specified in URI format, i.e. `tcp://localhost:9001`. - On supported platforms, the admin socket can listen on a UNIX domain socket instead, i.e. `unix:///var/run/yggdrasil.sock`. @@ -226,7 +226,7 @@ Note that any field not specified in the configuration will use its default valu - Each entry should begin with `tcp://` or `socks://proxyAddress:proxyPort/`. - `InterfacePeers` - Like peers above, but arranged using specific interface names: `{ "eth0": [ "tcp://peerAddress:peerPort", "tls://peerAddress:peerPort", "socks://proxyAddress:proxyPort/peerAddress:peerPort", ... ], "eth1": [], ... }` of peers to connect to. - - A `tcp://` peer connection can only connect to a `tcp://` listener, and a `tls://` peer connection can only connect to a `tls://` listener + - Note that a `tcp://` peer connection can only connect to a `tcp://` listener, and a `tls://` peer connection can only connect to a `tls://` listener - `AllowedEncryptionPublicKeys` - A list of strings in the form `["key", "key", ...]`, where `key` is each node's `EncryptionPublicKey` key which you would like to allow connections from. - This option allows you to restrict which other nodes can connect to your Yggdrasil node as a peer. It applies to incoming TCP connections. From 5154d94df66e4b04b84f0d5a126d5f16d2ec1b83 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 25 Oct 2019 13:49:55 +0100 Subject: [PATCH 62/95] Update configuration.md --- configuration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/configuration.md b/configuration.md index 41fbc59..e7a563e 100644 --- a/configuration.md +++ b/configuration.md @@ -294,17 +294,17 @@ Note that any field not specified in the configuration will use its default valu - `Enable` - Enables crypto-key routing. - If enabled, the following crypto-key routes will be used by Yggdrasil. If disabled, the below options have no effect. - - `IPv6Destinations` + - `IPv6RemoteSubnets` - A list of routes in the form `{ "aaaa:bbbb:cccc::/e": "EncryptionPublicKey", ... }` - For each entry, an IPv6 route entry will be created that sends traffic destined for `aaaa:bbbb:cccc::/e` to the node with the specified `EncryptionPublicKey` (effectively your "remote" ranges). - - `IPv6Sources` + - `IPv6LocalSubnets` - A list of allowed source subnets in the form `[ "aaaa:bbbb:cccc::/e" ]` - Specifies a list of source IPv6 addresses which are allowed to be sent over the tunnel (essentially your "local" ranges). - Traffic from the Yggdrasil node's IPv6 address and routed subnet are always allowed. - - `IPv4Destinations` + - `IPv4RemoteSubnets` - A list of routes in the form `{ "a.b.c.d/e": "EncryptionPublicKey", ... }` - For each entry, an IPv4 route entry will be created that sends traffic destined for `a.b.c.d/e` to the node with the specified `EncryptionPublicKey` (effectively your "remote" ranges). - - `IPv4Sources` + - `IPv4LocalSubnets` - A list of allowed source subnets in the form `[ "a.b.c.d/e" ]` - Specifies a list of source IPv4 addresses which are allowed to be sent over the tunnel (essentially your "local" ranges). - `SwitchOptions` From fbc16787c46849558ffb95e92100156d07da8f48 Mon Sep 17 00:00:00 2001 From: Arceliar Date: Sun, 3 Nov 2019 14:04:46 -0600 Subject: [PATCH 63/95] update genkeys part of configuration.md --- configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration.md b/configuration.md index e7a563e..f8725b3 100644 --- a/configuration.md +++ b/configuration.md @@ -383,7 +383,7 @@ This can partially mitigate the fact that IPv6 addresses are only 128 bits long, In short, if you plan to advertise a prefix, or if you want your address to be exceptionally difficult to collide with, then it is strongly advised that you burn some CPU cycles generating a harder-to-collide set of keys, using the following tool: ``` -GOPATH=$PWD go run -tags debug misc/genkeys.go +go run misc/genkeys.go ``` This continually generates new keys and prints them out each time a new best set of keys is discovered. From 71ec4e47da6f70eb9e1662893a769c61c0a3a894 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sun, 3 Nov 2019 21:45:34 +0000 Subject: [PATCH 64/95] Update 2018-11-06-crypto-key-routing.md --- _posts/2018-11-06-crypto-key-routing.md | 30 ++++++++++++------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/_posts/2018-11-06-crypto-key-routing.md b/_posts/2018-11-06-crypto-key-routing.md index a49dac2..6c3b2e9 100644 --- a/_posts/2018-11-06-crypto-key-routing.md +++ b/_posts/2018-11-06-crypto-key-routing.md @@ -33,25 +33,25 @@ is released: TunnelRouting: { Enable: false - IPv6Destinations: {} - IPv6Sources: [] - IPv4Destinations: {} - IPv4Sources: [] + IPv6RemoteSubnets: {} + IPv6LocalSubnets: [] + IPv4RemoteSubnets: {} + IPv4LocalSubnets: [] } ``` -The `IPv6Destinations` and `IPv4Destinations` options are used to specify +The `IPv6RemoteSubnets` and `IPv4RemoteSubnets` options are used to specify crypto-key routes. The subnet refers to an encryption public key of another node, for example: ``` -IPv6Destinations: { +IPv6RemoteSubnets: { "fd64:642b:1a20::/48": "ef78da7fc983c6c210609529921a701ca3e43fa5cfd79f5f20cc67bf66e45c1a", "fd25:8a33:9311:a53b::/64": "417fd0a66a104f050ae3544b3bc03eeb3648dded4a8c1fb085d65ffa25e83d6e" } ``` As with a typical routing table, more specific routes are preferred. -The `IPv6Sources` and `IPv4Sources` options are used to specify which source +The `IPv6LocalSubnets` and `IPv4LocalSubnets` options are used to specify which source addresses are eligible to be routed across a tunnel. For IPv6, this is optional as each Yggdrasil node already has an address and a routed subnet, and these are always allowed as source addresses. Specifying additional source addresses means @@ -63,32 +63,32 @@ mandatory as Yggdrasil nodes do not have IPv4 addresses by default. Assume that node A is `a.a.a.a/24` and `aaaa:aaaa:aaaa:aaaa::/64` and node B is `b.b.b.b/24` and `bbbb:bbbb:bbbb:bbbb::/64`. On node A, use the following `TunnelRouting` configuration: ``` Enable: true -IPv6Destinations: { +IPv6RemoteSubnets: { "bbbb:bbbb:bbbb:bbbb::/64": "xxxxxxxxxxxxxx" } -IPv6Sources: { +IPv6LocalSubnets: { aaaa:aaaa:aaaa:aaaa::/64 } -IPv4Destinations: { +IPv4RemoteSubnets: { "b.b.b.b/24": "xxxxxxxxxxxxxx" } -IPv4Sources: { +IPv4LocalSubnets: { a.a.a.a/24 } ``` On node B, use the reverse `TunnelRouting` configuration: ``` Enable: true -IPv6Destinations: { +IPv6RemoteSubnets: { "aaaa:aaaa:aaaa:aaaa::/64": "xxxxxxxxxxxxxx" } -IPv6Sources: { +IPv6LocalSubnets: { bbbb:bbbb:bbbb:bbbb::/64 } -IPv4Destinations: { +IPv4RemoteSubnets: { "a.a.a.a/24": "xxxxxxxxxxxxxx" } -IPv4Sources: { +IPv4LocalSubnets: { b.b.b.b/24 } ``` From ac1d07c9442f3894a42184dc99d4052b50d98721 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sun, 3 Nov 2019 21:46:33 +0000 Subject: [PATCH 65/95] Update 2018-11-06-crypto-key-routing.md --- _posts/2018-11-06-crypto-key-routing.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/_posts/2018-11-06-crypto-key-routing.md b/_posts/2018-11-06-crypto-key-routing.md index 6c3b2e9..1f3ac64 100644 --- a/_posts/2018-11-06-crypto-key-routing.md +++ b/_posts/2018-11-06-crypto-key-routing.md @@ -52,11 +52,7 @@ IPv6RemoteSubnets: { As with a typical routing table, more specific routes are preferred. The `IPv6LocalSubnets` and `IPv4LocalSubnets` options are used to specify which source -addresses are eligible to be routed across a tunnel. For IPv6, this is optional -as each Yggdrasil node already has an address and a routed subnet, and these are -always allowed as source addresses. Specifying additional source addresses means -that you can use the tunnel to route between networks. For IPv4, this is -mandatory as Yggdrasil nodes do not have IPv4 addresses by default. +addresses are eligible to be routed across a tunnel. These options are mandatory. ### Bridging networks From 88bfe32f2ad27888b169ecf155e58341cd25ee53 Mon Sep 17 00:00:00 2001 From: Dmitrii <8805785+mflsim@users.noreply.github.com> Date: Sun, 10 Nov 2019 16:10:36 +0300 Subject: [PATCH 66/95] Update services.md --- services.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/services.md b/services.md index b837fd7..21c0fe1 100644 --- a/services.md +++ b/services.md @@ -26,6 +26,9 @@ The following services are available on the Yggdrasil network, courtesy of our c - [Mesh Forum](http://[303:60d4:3d32:a2b9::4]/) - a chit-chat forum about everything. Run by @pztrn. - Accessible via Yggdrasil at `http://[303:60d4:3d32:a2b9::4]/`. + +- [Yggedit](http://[301:b614:c68e:b27f::1]/) - a reddit like social network/forum + - Accessible via Yggdrasil at `http://[301:b614:c68e:b27f::1]/`. ### Wikis From ec51db4aaacdd6f0f3e363db1d2c896c944af5c7 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sun, 24 Nov 2019 09:52:16 +0000 Subject: [PATCH 67/95] Update changelog.md --- changelog.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/changelog.md b/changelog.md index 903ffc7..a511c52 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,23 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> +## [0.3.12] - 2019-11-24 +### Added +- New API functions `SetMaximumSessionMTU` and `GetMaximumSessionMTU` +- New command line parameters `-address` and `-subnet` for getting the address/subnet from the config file, for use with `-useconffile` or `-useconf` +- A warning is now produced in the Yggdrasil output at startup when the MTU in the config is invalid or has been adjusted for some reason + +### Changed +- On Linux, outgoing `InterfacePeers` connections now use `SO_BINDTODEVICE` to prefer an outgoing interface +- The `genkeys` utility is now in `cmd` rather than `misc` + +### Fixed +- A data race condition has been fixed when updating session coordinates +- A crash when shutting down when no multicast interfaces are configured has been fixed +- A deadlock when calling `AddPeer` multiple times has been fixed +- A typo in the systemd unit file (for some Linux packages) has been fixed +- The NodeInfo and admin socket now report `unknown` correctly when no build name/version is available in the environment at build time +- The MTU calculation now correctly accounts for ethernet headers when running in TAP mode ## [0.3.11] - 2019-10-25 ### Added From e296fce994a7e0657fbed98507ea1fbb1a2f5079 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sun, 24 Nov 2019 09:52:30 +0000 Subject: [PATCH 68/95] Update index.md --- index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.md b/index.md index 2f48d77..c79e38c 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.11](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.12](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. From e180bdc7c2bdda577b87381cc134d3edd3a8837b Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 25 Nov 2019 11:22:31 +0000 Subject: [PATCH 69/95] Update AWDL posts, include OWLink URLs --- _posts/2019-08-19-awdl.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md index 327c169..4969a56 100644 --- a/_posts/2019-08-19-awdl.md +++ b/_posts/2019-08-19-awdl.md @@ -9,10 +9,13 @@ author: Neil Alexander I was mostly prompted to write this post in response to a [Hacker News thread](https://news.ycombinator.com/item?id=20735462) recently, which announced -the release of an open-source implementation of the Apple Wireless Direct Link -(AWDL) protocol. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay -and some other Apple wireless technologies. Even though everything covered in -this post was done some time ago, I have never spent the time to document it. +the release of an [open-source AirDrop implementation called +OpenDrop](https://github.com/seemoo-lab/opendrop) from the same team who +produced an [open-source implementation of Apple Wireless Direct Link (AWDL) +protocol called OWL](https://github.com/seemoo-lab/owl). AWDL is the secret +sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless +technologies. Even though everything covered in this post was done some time +ago, I have never spent the time to document it. With a few exceptions, most wireless networks in the world operate in "infrastructure mode" which is where a wireless access point serves one or more @@ -48,6 +51,13 @@ These "hops" between wireless channels happen so quickly that there's very little disruption to what the user is doing with their Wi-Fi connection already (except for some minor wireless performance degradation - to be covered later). +A number of papers have been published by the OWLink team on the inner workings +of the AWDL protocol, which can be [found +here](https://owlink.org/publications/). In particular, [this +paper](https://arxiv.org/pdf/1808.03156.pdf) from Mobicom 2018 contains a +significant amount of detail about the AWDL protocol itself, channel hopping +techniques and security considerations, amongst other things. + ### Mesh opportunities Yggdrasil is designed to create a mesh network automatically out of @@ -69,7 +79,8 @@ configuration options, and it should scale well too in principle. Yggdrasil can already discover potential peers on the same network segment by using multicast service discovery, which sounds a lot like what AWDL does on the -social channel. You can configure which interfaces Yggdrasil beacons on with the `MulticastInterfaces` configuration directive. +social channel. You can configure which interfaces Yggdrasil beacons on with the +`MulticastInterfaces` configuration directive. I wanted to know if we could blend the two so that Yggdrasil could automatically discover other nearby devices and initiate peering connections with them using From e5ee7afc1dcd92a2d1ebd509ef5c99b734cf26da Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 25 Nov 2019 11:24:14 +0000 Subject: [PATCH 70/95] Update AWDL posts, include OWLink URLs --- _posts/2019-08-19-awdl.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md index 4969a56..0f2ef3a 100644 --- a/_posts/2019-08-19-awdl.md +++ b/_posts/2019-08-19-awdl.md @@ -9,11 +9,11 @@ author: Neil Alexander I was mostly prompted to write this post in response to a [Hacker News thread](https://news.ycombinator.com/item?id=20735462) recently, which announced -the release of an [open-source AirDrop implementation called -OpenDrop](https://github.com/seemoo-lab/opendrop) from the same team who -produced an [open-source implementation of Apple Wireless Direct Link (AWDL) -protocol called OWL](https://github.com/seemoo-lab/owl). AWDL is the secret -sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless +the release of an open-source AirDrop implementation called +[OpenDrop](https://github.com/seemoo-lab/opendrop), from the same team at Seemoo +Lab who produced an open-source implementation of Apple Wireless Direct Link +(AWDL) protocol called [OWL](https://github.com/seemoo-lab/owl). AWDL is the +secret sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless technologies. Even though everything covered in this post was done some time ago, I have never spent the time to document it. From e86887bcaa74c19887dff2577494794a94e9e141 Mon Sep 17 00:00:00 2001 From: vkeb6rza Date: Wed, 27 Nov 2019 07:58:29 +0000 Subject: [PATCH 71/95] add vkeb6rza's public bitcoin services --- services.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/services.md b/services.md index 21c0fe1..4893db9 100644 --- a/services.md +++ b/services.md @@ -214,3 +214,13 @@ The following services are available on the Yggdrasil network, courtesy of our c - `iperf3 -6 -c y.thingylabs.io` on [ygg.thingylabs.io](https://ygg.thingylabs.io/) - Link: 10 GBit - Location: Nuremberg, Germany + +---- + +## Bitcoin + +### Node by [vkeb6rza](https://vkeb6rza.github.io/) at `204:381:4c98:df09:d15b:ab1e:894b:d99e` + - Port 8333 - Bitcoin Core with BIP157 and BIP158 experimental support + - Port 9735 - Lightning Network - `038758ca700b8c4c73d1b86440acb963be93e5b11f9d6f363041be43572c8cbd43@[204:381:4c98:df09:d15b:ab1e:894b:d99e]:9735` + - Port 9911 - LND Watchtower + - Port 50001 - Electrum server From ae946a5aa3ce8d9a94dadcf0c71fed9bbbbf89e0 Mon Sep 17 00:00:00 2001 From: Dimitris Apostolou Date: Fri, 29 Nov 2019 12:49:08 +0200 Subject: [PATCH 72/95] Fix typos --- _posts/2018-07-28-addressing.md | 2 +- _posts/2018-08-18-congestion-control.md | 6 +++--- _posts/2019-01-08-history.md | 2 +- _posts/2019-03-24-peering.md | 2 +- _posts/2019-08-19-awdl.md | 2 +- _posts/2019-09-01-actors.md | 4 ++-- admin.md | 2 +- changelog.md | 6 +++--- configuration.md | 2 +- 9 files changed, 14 insertions(+), 14 deletions(-) diff --git a/_posts/2018-07-28-addressing.md b/_posts/2018-07-28-addressing.md index a50c7a5..0183681 100644 --- a/_posts/2018-07-28-addressing.md +++ b/_posts/2018-07-28-addressing.md @@ -100,7 +100,7 @@ So full connection process looks something like the following: 5. The node checks that the destination `NodeID` and bitmask match the `NodeID` of the closest node (if not, then it means the destination node doesn't exist / is offline / is unreachable while the network re-converges due some disruption). 6. The node sends a session ping to the destination. 7. The node receives a session pong from the destination, learning their public ephemeral key. -8. The nodes can now send regular IPv6 traffic to eachother, encrypted with the ephemeral shared secret, using the session's cached `coords` to avoid future lookups (unless the session is unresponsive for too long, in which case any new sends will also trigger a ping, or a new DHT lookup if the pings fail). +8. The nodes can now send regular IPv6 traffic to each other, encrypted with the ephemeral shared secret, using the session's cached `coords` to avoid future lookups (unless the session is unresponsive for too long, in which case any new sends will also trigger a ping, or a new DHT lookup if the pings fail). ### Conclusion diff --git a/_posts/2018-08-18-congestion-control.md b/_posts/2018-08-18-congestion-control.md index d8a29ad..7d82362 100644 --- a/_posts/2018-08-18-congestion-control.md +++ b/_posts/2018-08-18-congestion-control.md @@ -27,7 +27,7 @@ This post attempts to explain Yggdrasil's congestion control mechanism, why past The first thing to try is not to implement any explicit buffering in Yggdrasil. Packets are received from a socket, we look up where the packet needs to go next, and then we send on that socket. -This immediately leads to blocking network operations and poor performance, so we need need separate read and write threads (goroutines, in our case). +This immediately leads to blocking network operations and poor performance, so we need separate read and write threads (goroutines, in our case). Initially, we used buffered channels and non-blocking channel sends. This means that, instead of the reader goroutine writing to the socket to send, it would pass it to a channel which a dedicated writer goroutine would read from. The problem with this approach is that Go channels with non-blocking sends are [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) and [tail dropped](https://en.wikipedia.org/wiki/Tail_drop). @@ -56,7 +56,7 @@ What we want is for multiple streams of traffic to be handled independently, to Then, we can reward different traffic streams to prioritize based on lowest bandwidth (i.e. size of queue / age of oldest packet in queue, with a separate queue per traffic stream). Now we let traffic streams compete for bandwidth. The winning strategy, to get more bandwidth during times of congestion, is to attempt to use *less* bandwidth, which I argue is exactly the behavior we want to encourage. -Streams of traffic that play nice get a fair share of bandwidth, which includes pretty much every sane TCP implementation, and streams that flood goto timeout. +Streams of traffic that play nice get a fair share of bandwidth, which includes pretty much every sane TCP implementation, and streams that flood go to timeout. ### Yggdrasil's congestion control @@ -99,7 +99,7 @@ Still, because we won't really know without trying, adding the required new pack Yggdrasil has gone through a number of different congestion control mechanisms since the TCP link layer was introduced. The current congestion control mechanism rewards traffic streams which utilize less bandwidth by prioritizing them higher than streams using more bandwidth. -Cooperative stream obtain a fair share of bandwidth, while stream which attempt to use more than their fair share are given lower priority, and are forced to throttle down as a result. +Cooperative streams obtain a fair share of bandwidth, while streams which attempt to use more than their fair share are given lower priority, and are forced to throttle down as a result. When packet drops become necessary, a random drop mechanism is used which penalizes large queues the most, which should signal congestion to the worst offenders. Much of this is a precursor to backpressure routing, which, if it works out in practice as well as it does on paper, should give the network a nearly-optimal latency/bandwidth trade-off. diff --git a/_posts/2019-01-08-history.md b/_posts/2019-01-08-history.md index 6bce200..31a8d3a 100644 --- a/_posts/2019-01-08-history.md +++ b/_posts/2019-01-08-history.md @@ -56,7 +56,7 @@ Then, when a node needs to forward a packet, it checks the tree location of each This is explained in more detail in earlier blog posts, if you're not familiar with how Yggdrasil routes and care to read more. In our package delivery example, imagine if the streets in Alice's town were laid out in a grid, and then named and numbered systematically by blocks, with street signs to label where any off-grid bypasses go. -Alice and friends still haven't bought maps, but they they know each other's *addresses* instead. +Alice and friends still haven't bought maps, but they know each other's *addresses* instead. So, if Alice wants to contact Carol, she first travels to Bob's house and asks him for Carol's address. Now, when she wants to deliver a package to Carol, she can simply follow the block structure of the town until she arrives on Carol's block, and she has the option to take any bypass she happens to come across if it brings her closer to Carol's place. That's basically how routing on the tree, or taking an off-tree shortcut, work in Yggdrasil's greedy routing scheme, except with a tree instead of a grid (which, in addition to working everywhere, seems to work *well* in the places we care about). diff --git a/_posts/2019-03-24-peering.md b/_posts/2019-03-24-peering.md index 2350c6e..b4764cb 100644 --- a/_posts/2019-03-24-peering.md +++ b/_posts/2019-03-24-peering.md @@ -28,7 +28,7 @@ In addition, the number of peers you want to add depends on what you want to do. ### What happens when things go wrong -Lets imagine we have some nodes in New York, and initially they follow the peering rules outlined above. Now suppose that two of these nodes decide that they want to add connections to London. In Yggdrasil, nodes tend to select parents that minimize latency to the root, which happens to be a node in Paris at the time I'm writing this. As a result, both of the NY nodes are likely to select their respective London peers as their parents. If the nodes are following the peering rules, then at least one of them has also decided to peer with the other, so they have a shortcut they can use to talk to each-other (or any descendants in the tree). +Let's imagine we have some nodes in New York, and initially they follow the peering rules outlined above. Now suppose that two of these nodes decide that they want to add connections to London. In Yggdrasil, nodes tend to select parents that minimize latency to the root, which happens to be a node in Paris at the time I'm writing this. As a result, both of the NY nodes are likely to select their respective London peers as their parents. If the nodes are following the peering rules, then at least one of them has also decided to peer with the other, so they have a shortcut they can use to talk to each-other (or any descendants in the tree). However, if they ignore the peering rules and *don't* peer with each other, then they are likely to route through London instead of communicating over their local mesh network. A shorter path exists, through their local mesh network, but it's not one that the network *must* know about for routing to work, so they won't necessarily know about it. As a result, the latency between these two nodes (or decedents thereof) will likely be an order of magnitude more than it needs to be (and probably lower bandwidth as well). diff --git a/_posts/2019-08-19-awdl.md b/_posts/2019-08-19-awdl.md index 0f2ef3a..94344dc 100644 --- a/_posts/2019-08-19-awdl.md +++ b/_posts/2019-08-19-awdl.md @@ -204,7 +204,7 @@ particular density due to only having a limited number of Macs to hand. One thing that I did notice though is that, while AWDL is active, my wireless connection to my home Wi-Fi network does reduce in speed somewhat. This is to be expected, given that the wireless chipset is hopping between channels rather -than spending all of it's time on a single channel. +than spending all of its time on a single channel. Sadly we weren't able to reproduce this test using iOS Testflight builds of Yggdrasil. On iOS, we implement Yggdrasil as a VPN service which is subject to a diff --git a/_posts/2019-09-01-actors.md b/_posts/2019-09-01-actors.md index d5b95d7..ad0bee4 100644 --- a/_posts/2019-09-01-actors.md +++ b/_posts/2019-09-01-actors.md @@ -52,7 +52,7 @@ Different implementations differ on details after that, such as what order messa -I'm particularly fond of the [pony](https://ponylang.io) programming language's take on the actor model. I really can't being to say enough nice things about their approach, and fully describing it is beyond the scope of this blog post, but if you come out of here with an interest in the actor model, then I highly recommend checking out that language. Maybe watch a few of the talks from the developers that have been posted to youtube, or read their papers about what is *easily* the most promising approach to garbage collection I've ever come across. +I'm particularly fond of the [pony](https://ponylang.io) programming language's take on the actor model. I really can't say enough nice things about their approach, and fully describing it is beyond the scope of this blog post, but if you come out of here with an interest in the actor model, then I highly recommend checking out that language. Maybe watch a few of the talks from the developers that have been posted to YouTube, or read their papers about what is *easily* the most promising approach to garbage collection I've ever come across. Anyway, I don't actually work on anything written in pony, but I like their version of the actor model so much that I decided to see if I could trick Go's runtime into faking it. The result is [`phony`](https://github.com/Arceliar/phony), which manages to do most of what I want in under 70 lines of code. When we write code using this asynchronous message passing style, instead of ordinary goroutines+channels, the implications are pretty significant: @@ -163,7 +163,7 @@ And that's about it. The first argument to `myActor.RunTheFunction` also `nil`ab What's great is that we don't need to think about starting or stopping workers, deadlocks and leaks are not possible outside of blocking operations (e.g. I/O), and we can add or reuse behaviors just as easily as any function. I find the code easier to read and reason about too. -I/O is one rough spot, since an `Actor` can block on a `Read` or a `Write` and not process incoming messages as a result. This isn't really any worse than working with normal Go code, and the pattern we've adopted is to have separate `Actor`s for `Read` and `Write`, where one mostly just sits in a `Read` loop and sends the results (and/or error) somewhere whenever a `Read` finishes. These two workers can be children of some parent `Actor`, which is the only one the rest of the code needs to know about, and then all we need to remember to do is close the `ReadWriteCloser` (e.g. socket) at some point when we're done. This is the sort of thing that we'll eventually want to write a standard `struct` for, update our code everywhere to use it, and then never have to think about it again. In the mean time, we have a couple of very similar implementations for working with sockets or the tun/tap device. +I/O is one rough spot, since an `Actor` can block on a `Read` or a `Write` and not process incoming messages as a result. This isn't really any worse than working with normal Go code, and the pattern we've adopted is to have separate `Actor`s for `Read` and `Write`, where one mostly just sits in a `Read` loop and sends the results (and/or error) somewhere whenever a `Read` finishes. These two workers can be children of some parent `Actor`, which is the only one the rest of the code needs to know about, and then all we need to remember to do is close the `ReadWriteCloser` (e.g. socket) at some point when we're done. This is the sort of thing that we'll eventually want to write a standard `struct` for, update our code everywhere to use it, and then never have to think about it again. In the meantime, we have a couple of very similar implementations for working with sockets or the tun/tap device. ### Dénouement diff --git a/admin.md b/admin.md index c28cc70..c18536b 100644 --- a/admin.md +++ b/admin.md @@ -252,7 +252,7 @@ Returns: #### `removeRoute` Expects: -- `subnet=` `string` for the subnet to remove the route route for +- `subnet=` `string` for the subnet to remove the route for - `box_pub_key=` `string` for the public key that is routed to Removes an existing crypto-key route. diff --git a/changelog.md b/changelog.md index a511c52..8b8ec85 100644 --- a/changelog.md +++ b/changelog.md @@ -66,7 +66,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ### Changed - On recent Linux kernels, Yggdrasil will now set the `tcp_congestion_control` algorithm used for its own TCP sockets to [BBR](https://github.com/google/bbr), which reduces latency under load -- The systemd service configuration in `contrib` (and, by extension, some of our packages) now attemps to load the `tun` module, in case TUN/TAP support is available but not loaded, and it restricts Yggdrasil to the `CAP_NET_ADMIN` capability for managing the TUN/TAP adapter, rather than letting it do whatever the (typically `root`) user can do +- The systemd service configuration in `contrib` (and, by extension, some of our packages) now attempts to load the `tun` module, in case TUN/TAP support is available but not loaded, and it restricts Yggdrasil to the `CAP_NET_ADMIN` capability for managing the TUN/TAP adapter, rather than letting it do whatever the (typically `root`) user can do ### Fixed - The `yggdrasil.Conn.RemoteAddr()` function no longer blocks, fixing a deadlock when CKR is used while under heavy load @@ -180,7 +180,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ## [0.3.4] - 2019-03-12 ### Added - Support for multiple listeners (although currently only TCP listeners are supported) -- New multicast behaviour where each multicast interface is given it's own link-local listener and does not depend on the `Listen` configuration +- New multicast behaviour where each multicast interface is given its own link-local listener and does not depend on the `Listen` configuration - Blocking detection in the switch to avoid parenting a blocked peer - Support for adding and removing listeners and multicast interfaces when reloading configuration during runtime - Yggdrasil will now attempt to clean up UNIX admin sockets on startup if left behind by a previous crash @@ -374,7 +374,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - Wire format changes (backwards incompatible). - Less maintenance traffic per peer. - Exponential back-off for DHT maintenance traffic (less maintenance traffic for known good peers). -- Iterative DHT (added some time between v0.1.0 and here). +- Iterative DHT (added sometime between v0.1.0 and here). - Use local queue sizes for a sort of local-only backpressure routing, instead of the removed bandwidth estimates, when deciding where to send a packet. ### Removed diff --git a/configuration.md b/configuration.md index f8725b3..2fd2994 100644 --- a/configuration.md +++ b/configuration.md @@ -371,7 +371,7 @@ interface eth0 ``` Note that a `/64` prefix has fewer bits of address space available to check against the node's ID, which in turn means hash collisions are more likely. -As such, it is unwise to rely on addresses as a form of identify verification for the `300::/8` address range. +As such, it is unwise to rely on addresses as a form of identity verification for the `300::/8` address range. ## Generating Stronger Addresses (and Prefixes) From e53547964c5eebcb6e1f78b2d84c2952d1012667 Mon Sep 17 00:00:00 2001 From: Maxim Reznik Date: Sat, 30 Nov 2019 13:50:30 +0200 Subject: [PATCH 73/95] Fix path to genkeys.go It seems file has been moved --- configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration.md b/configuration.md index f8725b3..8aa539e 100644 --- a/configuration.md +++ b/configuration.md @@ -383,7 +383,7 @@ This can partially mitigate the fact that IPv6 addresses are only 128 bits long, In short, if you plan to advertise a prefix, or if you want your address to be exceptionally difficult to collide with, then it is strongly advised that you burn some CPU cycles generating a harder-to-collide set of keys, using the following tool: ``` -go run misc/genkeys.go +go run cmd/genkeys/main.go ``` This continually generates new keys and prints them out each time a new best set of keys is discovered. From 2b9c3485a8a15eaa6a0d76fa305593cdfd78fc8c Mon Sep 17 00:00:00 2001 From: Peter Gervai Date: Mon, 2 Dec 2019 10:42:12 +0100 Subject: [PATCH 74/95] Fix configuration doc 'yggdrasilctl' references to 'yggdrasil', fix '--param' format --- configuration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/configuration.md b/configuration.md index f8725b3..3d33644 100644 --- a/configuration.md +++ b/configuration.md @@ -23,15 +23,15 @@ Otherwise, you can generate a configuration file in the following ways: Yggdrasil can accept a configuration file either through `stdin` or by being given a path on the filesystem to a configuration file: -- **Using stdin**: `yggdrasilctl --useconf < /etc/yggdrasil.conf` -- **Using file:** `yggdrasilctl --useconffile /etc/yggdrasil.conf` +- **Using stdin**: `yggdrasil -useconf < /etc/yggdrasil.conf` +- **Using file:** `yggdrasil -useconffile /etc/yggdrasil.conf` ## Normalising Configuration If you want to see the original format of the configuration file, or convert between HJSON and JSON formats, you can use the `-normaliseconf` option, e.g. -- **Convert from HJSON to JSON**: `yggdrasilctl -normaliseconf -useconffile /etc/yggdrasil.conf -json` -- **Convert from JSON to HJSON**: `yggdrasilctl -normaliseconf -useconffile /etc/yggdrasil.conf` +- **Convert from HJSON to JSON**: `yggdrasil -normaliseconf -useconffile /etc/yggdrasil.conf -json` +- **Convert from JSON to HJSON**: `yggdrasil -normaliseconf -useconffile /etc/yggdrasil.conf` Normalising the configuration also adds any missing configuration items with their default values. This can be useful when upgrading to a newer version of Yggdrasil that adds new configuration options. Many of our distribution packages normalise the configuration automatically during upgrade. From a6de98f8b0419af47ff1d590f965000b92b95fb3 Mon Sep 17 00:00:00 2001 From: George <57254463+zhoreeq@users.noreply.github.com> Date: Tue, 17 Dec 2019 16:02:12 -0500 Subject: [PATCH 75/95] Add chat.station.moe XMPP server --- services.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/services.md b/services.md index 4893db9..f1149f7 100644 --- a/services.md +++ b/services.md @@ -89,6 +89,13 @@ The following services are available on the Yggdrasil network, courtesy of our c ## XMPP +### chat.station.moe + +- `chat.station.moe`, run by [zhoreeq](https://github.com/zhoreeq) + - Webchat is at [http://[305:1bba:6a3:7f24::23]/](http://[305:1bba:6a3:7f24::23]/) or http://chat.station.moe/. General chat room is at `ygg@conference.station.moe`. + - Unnecessary TLS is removed. XMPP clients should allow authorization over plaintext and disable TLS/SSL/STARTTLS enforcement. + - XMPP federation is enabled inside Yggdrasil network. [Example config file for Prosody](https://gist.github.com/zhoreeq/0f293cc7c338554369261cfdfab55ddc). + ### netwhood.online - `netwhood.online`, run by [abslimit](http://netwhood.online/feedback/) From 5cc6bf7aa3dac57136ad9b7178aae2b30b3a01c9 Mon Sep 17 00:00:00 2001 From: Kyler Chin <7539174+kylerchin@users.noreply.github.com> Date: Wed, 22 Jan 2020 08:24:18 -0800 Subject: [PATCH 76/95] Add Yggdrasil IPFS Bootstrap Node --- services.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/services.md b/services.md index f1149f7..3499a50 100644 --- a/services.md +++ b/services.md @@ -148,6 +148,11 @@ The following services are available on the Yggdrasil network, courtesy of our c - [neilalexander](https://matrix.to/#/@neilalexander:matrix.org)'s IPFS gateway is available at `http://ipfs.y.neilalexander.eu/ipfs/` ([test link](http://ipfs.y.neilalexander.eu/ipfs/QmZiSAYkU7gZtqYeZWL21yuwgFtRnJu1JjDzR6Qd2qdDBr/)) +### IPFS Bootstrap Swarm + +- Irvine, CA Based Server Boostrap Node hosted by @KylerChin +`/ip6/202:d0ca:a9d7:b4e8:bd3c:ffde:5c89:a3d7/tcp/4001/ipfs/QmZEiPvrfZHapq4uiyTDEcR2szCUhDnjdS4q3Uv2b1Uh88` + ---- ## Gaming From 4a64567728c2aae47bd01df8794b3b5a971e4afe Mon Sep 17 00:00:00 2001 From: Rany <31795451+rany0@users.noreply.github.com> Date: Thu, 30 Jan 2020 22:02:30 +0200 Subject: [PATCH 77/95] fix fedora installation error `sudo tee` will run as root `sudo cat` doesn't do anything you would get this error: ``` [rany@HPipi ~]$ sudo cat > /etc/yum.repos.d/yggdrasil.repo << EOF > [yggdrasil] > name = Yggdrasil > baseurl = https://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/rpm/ > gpgcheck=1 > gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-yggdrasil > EOF bash: /etc/yum.repos.d/yggdrasil.repo: Permission denied ``` --- installation-linux-rpm.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/installation-linux-rpm.md b/installation-linux-rpm.md index 12a55ba..d5a864e 100644 --- a/installation-linux-rpm.md +++ b/installation-linux-rpm.md @@ -21,13 +21,11 @@ gpg --armor --no-comment --export-options export-minimal --export 569130E8CA20FB Add the repository: ``` -sudo cat > /etc/yum.repos.d/yggdrasil.repo << EOF -[yggdrasil] +echo '[yggdrasil] name = Yggdrasil baseurl = https://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/rpm/ gpgcheck=1 -gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-yggdrasil -EOF +gpgkey=file:///etc/pki/rpm-gpg/RP^CGPG-KEY-yggdrasil' | sudo tee /etc/yum.repos.d/yggdrasil.repo ``` Create the `yggdrasil` group on your system: From 13f54fb85db6b59fa99d2d1b36583ec7ed52f1c1 Mon Sep 17 00:00:00 2001 From: Rany <31795451+rany0@users.noreply.github.com> Date: Thu, 30 Jan 2020 22:09:45 +0200 Subject: [PATCH 78/95] fix gpg name --- installation-linux-rpm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/installation-linux-rpm.md b/installation-linux-rpm.md index d5a864e..c0df9cb 100644 --- a/installation-linux-rpm.md +++ b/installation-linux-rpm.md @@ -25,7 +25,7 @@ echo '[yggdrasil] name = Yggdrasil baseurl = https://neilalexander.s3.dualstack.eu-west-2.amazonaws.com/rpm/ gpgcheck=1 -gpgkey=file:///etc/pki/rpm-gpg/RP^CGPG-KEY-yggdrasil' | sudo tee /etc/yum.repos.d/yggdrasil.repo +gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-yggdrasil' | sudo tee /etc/yum.repos.d/yggdrasil.repo ``` Create the `yggdrasil` group on your system: From 79657cb2adcc8ab5b4d00ecabe65b066f03d9b19 Mon Sep 17 00:00:00 2001 From: kaotisk Date: Tue, 4 Feb 2020 17:29:27 +0200 Subject: [PATCH 79/95] Added my irc network --- services.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/services.md b/services.md index f1149f7..4dfcecd 100644 --- a/services.md +++ b/services.md @@ -85,6 +85,14 @@ The following services are available on the Yggdrasil network, courtesy of our c - Yggdrasil's SSL port uses same certificate as the public Internet node, so you **will** receive SSL warning, this is expected - Join `#en` for english discussions +### ED12 IRC + +- `y.kaotisk-hund.tk` is accessible via Yggdrasil at `202:68fd:1f72:8505:74d6:b751:29ce:b5bb` port `6667` (TCP) + +- Main channel is `#general` +- Other channels can be listed with description as their topic +- Also accessible via cjdns at `fc42:7cfa:b830:e988:f192:717f:6576:ed12` port `6667` (TCP) + ---- ## XMPP From 269474ba826ac8ce782429938281f5836cebd04f Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 17 Feb 2020 20:15:50 +0000 Subject: [PATCH 80/95] Update for Windows documentation --- index.md | 2 +- installation-windows.md | 96 ++++++++++++++++++++++------------------- installation.md | 2 +- platform-windows.md | 19 -------- platforms.md | 4 -- 5 files changed, 53 insertions(+), 70 deletions(-) delete mode 100644 platform-windows.md diff --git a/index.md b/index.md index c79e38c..9a82b36 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.12](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.13](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. diff --git a/installation-windows.md b/installation-windows.md index 0aa0a57..b3de26e 100644 --- a/installation-windows.md +++ b/installation-windows.md @@ -3,61 +3,67 @@ tags: dontlink sitemap: true --- -# Installing manually on Windows +# Installing using the Windows installer -Yggdrasil is supported on Windows. You can either [download the latest binary from CircleCI](builds.md). +Yggdrasil is supported on Windows. You can [download the latest installer from +GitHub](https://github.com/yggdrasil-network/yggdrasil-go/releases). -## TAP Driver +## TUN driver -Yggdrasil depends on the OpenVPN TAP driver to work on Windows. There are two flavours of this driver: +Starting with version 0.3.13, Yggdrasil on Windows uses the Wireguard TUN +driver. If this is not installed on the system already, it will be installed +automatically by the Yggdrasil installer. -- [NDIS 5](https://swupdate.openvpn.org/community/releases/tap-windows-9.9.2_3.exe) (`tap-windows-9.9.2_3`) - recommended -- [NDIS 6](https://swupdate.openvpn.org/community/releases/tap-windows-9.21.2.exe) (`tap-windows-9.21.2`) - functional +For this reason, it is important that you use the correct installer for your +architecture - make sure to use the `x64` installer on 64-bit Windows and the +`x86` installer on 32-bit Windows. -Yggdrasil works with either driver, although the performance is remarkably better with the older NDIS 5 driver, therefore it is recommended to use that instead of the NDIS 6 driver. +Please note that the OpenVPN TAP driver is **no longer supported**. -Please note that if you already have OpenVPN for Windows, you likely have one of the two drivers installed already and should not install it again. +Once Yggdrasil is started, a new virtual network adapter will be created called +`Yggdrasil` by default, although this can be renamed using the `IfName` option +in the configuration file (below). The virtual network adapter will *not* be +visible on the system when Yggdrasil is not running. -## Installation +## Configuration -Start by [downloading the latest Yggdrasil binary](builds.md) and use Windows Explorer to copy it into a sensible location and rename it to `yggdrasil.exe`. +The Yggdrasil installer will automatically generate an `yggdrasil.conf` +configuration file, if one does not exist, in the following locations: -The below examples assume that you placed `yggdrasil.exe` into `C:\Program Files\Yggdrasil`. +- 64-bit Windows: `C:\Program Files\Yggdrasil` +- 32-bit Windows: `C:\Program Files (x86)\Yggdrasil` -## Generate configuration +## Windows Service -Before starting Yggdrasil, you should generate configuration. Open a Command Prompt as Administrator: +Yggdrasil is installed as a Windows service that starts automatically with +Windows. You can start, stop and restart Yggdrasil using the Services MMC +snap-in (`services.msc`) or, in more recent versions of Windows, the "Services" +tab in Task Manager. + +You will need to restart the Yggdrasil service after each change to the +configuration file. + +## Windows Firewall + +Windows Firewall may prompt for the Yggdrasil process to allow incoming or +outgoing connections. If so, you should allow this or Yggdrasil may not be able +to establish peerings correctly. + +Note that Yggdrasil, by default, does allow incoming traffic over the +`Yggdrasil` virtual adapter, therefore you may wish to designate the `Yggdrasil` +virtual network adapter as a public network in the Windows Firewall so that +unexpected incoming connections are blocked automatically. + +If you do, pay particular attention to ensure that file sharing (SMB), remote +procedure call (RPC) or remote desktop (RDP) services are not allowed on public +networks unless you explicitly need them to be accessible from remote Yggdrasil +hosts. + +## yggdrasilctl + +The `yggdrasilctl` utility is also installed into the same location above. You +can query Yggdrasil's runtime state using this tool using a Command Prompt or +PowerShell command line, e.g. ``` -"C:\Program Files\Yggdrasil\yggdrasil.exe" -genconf > "C:\Program Files\Yggdrasil\yggdrasil.conf" +"C:\Program Files\Yggdrasil\yggdrasilctl.exe" getPeers ``` - -## Run Yggdrasil - -### Run once - -Open a Command Prompt as Administrator and start Yggdrasil using your generated configuration: -``` -"C:\Program Files\Yggdrasil\yggdrasil.exe" -useconffile "C:\Program Files\Yggdrasil\yggdrasil.conf" -``` -Alternatively, start Yggdrasil in auto-configuration mode: -``` -"C:\Program Files\Yggdrasil\yggdrasil.exe" -autoconf -``` - -### Run as a background service - -Running as a background system service means that Yggdrasil will automatically start up in the background when Windows boots. - -Assuming that Yggdrasil is installed into `C:\Program Files\Yggdrasil` and your configuration *already exists* in `C:\Program Files\Yggdrasil\yggdrasil.conf`, as above, then you can install Yggdrasil as a Windows service. Open a Command Prompt as Administrator: -``` -sc create yggdrasil binpath= "\"C:\Program Files\Yggdrasil\yggdrasil.exe\" -useconffile \"C:\Program Files\Yggdrasil\yggdrasil.conf\"" -sc config yggdrasil displayname= "Yggdrasil Service" -sc config yggdrasil start= "auto" -sc start yggdrasil -``` -Alternatively, if you want the service to autoconfigure instead of using an `yggdrasil.conf`, replace the `sc create` line from above with: -``` -sc create yggdrasil binpath= "\"C:\Program Files\Yggdrasil\yggdrasil.exe\" -autoconf" -``` - -The Yggdrasil service can then be stopped and started using `services.msc`, or in more recent versions of Windows, the Task Manager. diff --git a/installation.md b/installation.md index cced1eb..d1abc02 100644 --- a/installation.md +++ b/installation.md @@ -24,4 +24,4 @@ Installation guides are available for the following platforms/distributions. ## Windows -- [Manual install as a Windows service](installation-windows.md) +- [Windows `.msi` installer](installation-windows.md) diff --git a/platform-windows.md b/platform-windows.md deleted file mode 100644 index 626fc2e..0000000 --- a/platform-windows.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -tags: dontlink -sitemap: true ---- - -# Windows - -Yggdrasil has been tested and is working on the following versions of Windows: - -- Windows 7 -- Windows 8.1 -- Windows 10 -- Windows Server 2016 - -It may work on other versions of Windows too, although these are untested. - -## Notes - -- TUN mode is not supported on Windows, and attempts to configure TUN mode will default to TAP mode automatically. diff --git a/platforms.md b/platforms.md index eed9c31..9b97067 100644 --- a/platforms.md +++ b/platforms.md @@ -16,10 +16,6 @@ sitemap: true - See the [iOS platform page](platform-ios.md). -## Windows - -- See the [Windows platform page](platform-windows.md). - ## EdgeRouter - See the [EdgeRouter platform page](platform-edgerouter.md). From 30bbb91cbfcf31d2c12e62d2982b481ed1739d68 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 17 Feb 2020 20:17:04 +0000 Subject: [PATCH 81/95] Update gems --- Gemfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Gemfile b/Gemfile index f419341..51cbf65 100644 --- a/Gemfile +++ b/Gemfile @@ -7,3 +7,5 @@ gem "jekyll-feed", "~> 0.11.0" gem "jekyll-sitemap", "~> 1.2" gem "github-pages", group: :jekyll_plugins + +gem "nokogiri", ">= 1.10.4" From 30b08f6eb2757607c87f10744a2f116ff11dbdc8 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Mon, 17 Feb 2020 22:55:25 +0000 Subject: [PATCH 82/95] Add blog post --- _posts/2020-02-21-release-v0-3-13.md | 121 +++++++++++++++++++++++++++ index.md | 2 +- 2 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 _posts/2020-02-21-release-v0-3-13.md diff --git a/_posts/2020-02-21-release-v0-3-13.md b/_posts/2020-02-21-release-v0-3-13.md new file mode 100644 index 0000000..623a808 --- /dev/null +++ b/_posts/2020-02-21-release-v0-3-13.md @@ -0,0 +1,121 @@ +--- +layout: post +title: "Release v0.3.11" +date: 2020-02-21 09:00:00 +0000 +author: Neil Alexander +--- + +### Release time! + +Our last Yggdrasil release, v0.3.12, was merged a couple of months ago at the +end of November. For the most part we have seen good stability with the v0.3.12 +builds, not to mention good adoption (with the crawler showing over 500 nodes +running it). Today we are releasing our next version, v0.3.13. + +Many of our releases tend not to warrant blog post entries, especially given +that the changelog documents the changes. However, there's some fairly big news +points associated with this version therefore this post aims to discuss them in +a bit more detail. + +#### TUN adapter changes + +The first big talking point is that this is the first Yggdrasil release that +departs entirely from the Water library and replaces it with the Wireguard TUN +library. There are a few reasons why we decided to switch from Water to the +Wireguard library, but one of the most prominent is that it gives us better TUN +support across all platforms and allows us to finally remove TAP support +altogether. + +At a high-level, TUN interfaces are effectively emulating "Layer 3" interfaces - +they deal only in IP packets - whereas TAP interfaces are emulating "Layer 2" +full-fat Ethernet interfaces. + +To run in TAP mode, Yggdrasil not only had to add and remove Ethernet headers +for each packet, but it also has to implement an entire NDP implementation and +track MAC addresses in order to trick the host operating system into believing +that there was a real Ethernet domain on the other end of the adapter. Needless +to say, the amount of boilerplate code in order to make TAP mode work correctly +was significant and much of that code was very fragile. + +Although we implemented NDP, we did not ever get around to implementing ARP, +which also meant that sending tunnel-routed IPv4 traffic over TAP interfaces +invariably did not work either. We have now been able to remove much of this +code and simplify the TUN code massively, closing the gaps between some of our +supported platforms. + +There is one platform that is negatively impacted by this change and that's +NetBSD. The Wireguard TUN package that we are using currently has **no support +for NetBSD**, so we are also removing NetBSD as a supported target until the +necessary code appears upstream. To our knowledge, we don't have a base of +NetBSD users anyway, but we will aim to re-add this soon. + +The `IfTAPMode` configuration option has now been removed from Yggdrasil +entirely and it will be ignored if specified. **If you are using TAP mode today, +then this will affect you**. Please make sure to check your Yggdrasil +configuration since this may result in interface naming changes and you may have +to update network settings in your host operating system. + +Initially we added TAP support into Yggdrasil as it was the only way to support +Windows, since the OpenVPN driver that we used at the time only supported TAP +mode. Thankfully, this is no longer a problem, as the Wireguard project have +also released [Wintun](https://wintun.net), which is supported by the Wireguard +TUN library. The net result is that we gain TUN support on Windows and the +performance is *far* better than the buggy OpenVPN driver, which is a nice segue +into... + +#### Windows installer and performance + +We have spent a lot of time trying to improve the installation and setup +experience on Windows. This mostly falls into two areas. + +The first is that using the Wintun driver has *massively* improved performance, +in some cases by hundreds of MB/s, and starting the Yggdrasil process is now +much more reliable too - it should no longer be necessary to restart Yggdrasil +due to cases of the TAP adapter not being set up or configured correctly. + +The second is that we now automatically generate Windows `.msi` installers using +Appveyor, which means that installing or upgrading Yggdrasil is now simpler than +ever. It is no longer necessary to create directories, copy files and register +Windows services by hand - a marked improvement! + +The installer also bundles the Wintun driver and it is installed automatically +if required, therefore there is no longer a need to hunt down and install the +OpenVPN TAP driver separately. We hope that these changes will help to encourage +adoption of Yggdrasil on Windows platforms by significantly reducing the barrier +to entry. + +As in the previous section, Yggdrasil on Windows has gone from supporting TAP +mode only to now supporting TUN mode only. **This may mean that you need to +review your configuration**. If you no longer need the OpenVPN TAP driver on +your system, it is best to entirely uninstall it. It is also important to make +sure that the `IfName` configuration option in your `yggdrasil.conf` does not +specify the same name as an existing OpenVPN TAP interface or Yggdrasil may fail +to start. + +#### End of the v0.3 release cycle + +Generally we try, where possible, to avoid make any changes which would damage +backward compatibility with previous versions. The last version that had +breaking changes was v0.2.1 - over a year and a half ago. However, maintaining +backward compatibility so tightly also prevents us from improving the Yggdrasil +design in various ways. + +Therefore, unless any serious bugs or security vulnerabilities appear, it is +very likely that this version will be the last in the v0.3 release cycle. +Instead, we will start working on the v0.4 release, which is likely to include a +number of breaking protocol changes and will be incompatible with v0.3 releases +as a result. + +More information will be announced on the types of changes in v0.4 as they +happen - expect to see more blog posts and chatter in the Matrix channel on this +subject - but we will aim to give as much notice as possible before releases +occur that contain breaking changes. + +#### Final mentions + +In addition to the release notes above, I'd like to relay the message that +[@mwarning](https://github.com/mwarning) has a proposal open for a Google Summer +of Code (GSoC) project under the Freifunk umbrella, comparing a number of mesh +routing protocols including Yggdrasil. More information about the proposal is +available [here](https://projects.freifunk.net/#/projects?project=freifunk_meshnet_protocol_evaluation&lang=en). +If you are interested, please reach out! diff --git a/index.md b/index.md index 9a82b36..d8c06e7 100644 --- a/index.md +++ b/index.md @@ -21,7 +21,7 @@ Binaries are [automatically built](https://circleci.com/gh/yggdrasil-network/ygg ## Implementation -Yggdrasil is written in Go. The codebase is fairly small and easy to navigate. It has been tested on a number of platforms (including [Linux](platform-linux.md), [Windows](platform-windows.md), [macOS](platform-macos.md), NetBSD, FreeBSD, OpenBSD and [Ubiquiti EdgeOS](platform-edgerouter.md)) although with various minor caveats. See the [Platforms](platforms.md) page for more information. +Yggdrasil is written in Go. The codebase is fairly small and easy to navigate. It has been tested on a number of platforms (including [Linux](platform-linux.md), [Windows](platform-windows.md), [macOS](platform-macos.md), FreeBSD, OpenBSD and [Ubiquiti EdgeOS](platform-edgerouter.md)) although with various minor caveats. See the [Platforms](platforms.md) page for more information. ## Get Involved From 0f132952ba27f7a3e974d5ae06e60303af1b6819 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Tue, 18 Feb 2020 18:47:41 +0000 Subject: [PATCH 83/95] Update 2020-02-21-release-v0-3-13.md --- _posts/2020-02-21-release-v0-3-13.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2020-02-21-release-v0-3-13.md b/_posts/2020-02-21-release-v0-3-13.md index 623a808..e4f2fac 100644 --- a/_posts/2020-02-21-release-v0-3-13.md +++ b/_posts/2020-02-21-release-v0-3-13.md @@ -1,6 +1,6 @@ --- layout: post -title: "Release v0.3.11" +title: "Release v0.3.13" date: 2020-02-21 09:00:00 +0000 author: Neil Alexander --- From 69d737b9b170f0ed115ede602064c21cd770284e Mon Sep 17 00:00:00 2001 From: George <57254463+zhoreeq@users.noreply.github.com> Date: Wed, 19 Feb 2020 13:24:55 -0500 Subject: [PATCH 84/95] Fix URL hover boldness, use color change instead URL boldness on hover is making the rest of text "jump" back and forth in Firefox --- assets/css/style.scss | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/assets/css/style.scss b/assets/css/style.scss index e64e727..01574b1 100644 --- a/assets/css/style.scss +++ b/assets/css/style.scss @@ -155,3 +155,8 @@ div.blogpost div#overview div#excerpt { text-decoration: none; font-size: 1em; } + +a:hover, a:focus { + color:#0be; + font-weight: initial; +} From 599cea47fec8e76d24b92e69c548aaef177568ee Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 21 Feb 2020 18:44:28 +0000 Subject: [PATCH 85/95] Update changelog.md --- changelog.md | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/changelog.md b/changelog.md index 8b8ec85..ab3e966 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,31 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> +## [0.3.13] - 2020-02-21 +### Added +- Support for the Wireguard TUN driver, which now replaces Water and provides far better support and performance on Windows +- Windows `.msi` installer files are now supported (bundling the Wireguard TUN driver) +- NodeInfo code is now actorised, should be more reliable +- The DHT now tries to store the two closest nodes in either direction instead of one, such that if a node goes offline, the replacement is already known +- The Yggdrasil API now supports dialing a remote node using the public key instead of the Node ID + +### Changed +- The `-loglevel` command line parameter is now cumulative and automatically includes all levels below the one specified +- DHT search code has been significantly simplified and processes rumoured nodes in parallel, speeding up search time +- DHT search results are now sorted +- The systemd service now handles configuration generation in a different unit +- The Yggdrasil API now returns public keys instead of node IDs when querying for local and remote addresses + +### Fixed +- The multicast code no longer panics when shutting down the node +- A potential OOB error when calculating IPv4 flow labels (when tunnel routing is enabled) has been fixed +- A bug resulting in incorrect idle notifications in the switch should now be fixed +- MTUs are now using a common datatype throughout the codebase + +### Removed +- TAP mode has been removed entirely, since it is no longer supported with the Wireguard TUN package. Please note that if you are using TAP mode, you may need to revise your config! +- NetBSD support has been removed until the Wireguard TUN package supports NetBSD + ## [0.3.12] - 2019-11-24 ### Added - New API functions `SetMaximumSessionMTU` and `GetMaximumSessionMTU` @@ -90,7 +115,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - Some minor memory leaks in the switch have been fixed, which improves memory usage on mobile builds - A memory leak in the add-peer loop has been fixed - The admin socket now reports the correct URI strings for SOCKS peers in `getPeers` -- A race condition when dialling a remote node by both the node address and routed prefix simultaneously has been fixed +- A race condition when dialing a remote node by both the node address and routed prefix simultaneously has been fixed - A race condition between the router and the dial code resulting in a panic has been fixed - A panic which could occur when the TUN/TAP interface disappears (e.g. during soft-shutdown) has been fixed - A bug in the semantic versioning script which accompanies Yggdrasil for builds has been fixed From e4903fab8ed511b96a5baa05111442e21eb71c10 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Fri, 21 Feb 2020 20:05:46 +0000 Subject: [PATCH 86/95] Update builds.md --- builds.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/builds.md b/builds.md index 2bcd579..ecf9458 100644 --- a/builds.md +++ b/builds.md @@ -9,6 +9,8 @@ After every push to the `master` branch of the [GitHub repository](https://githu If you prefer to live dangerously and want to test bleeding-edge features or changes, development builds are [available from the `develop` branch](builds-develop.md), although these may not be as stable. +Looking for Windows installers? [Click here to go to AppVeyor](https://ci.appveyor.com/project/neilalexander/yggdrasil-go/build/artifacts). + For convenience, the latest builds are linked below:

yggdrasil

From 1467a5ba74f33164a0a04f06ca62d963c3f1f985 Mon Sep 17 00:00:00 2001 From: Ellenor Malik Date: Fri, 21 Feb 2020 19:10:08 -0800 Subject: [PATCH 87/95] services.md update - new XMPP service with two hostnames --- services.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/services.md b/services.md index e54add5..9dfc44a 100644 --- a/services.md +++ b/services.md @@ -110,6 +110,12 @@ The following services are available on the Yggdrasil network, courtesy of our c - SRV record `_xmpp-client._tcp` is prioritized for `y.netwhood.online` (`202:12a9:e5:4474:d473:82be:16ac:9381`). XMPP-client must try yggdrasil address at first. - SRV record `_xmpp-server._tcp` is prioritized for `y.netwhood.online` too. XMPP s2s connections must try yggdrasil address at first. Lets peer? - "HTTP upload" XEP module is tied to `netwhood.online`, it will connect through the Internet. It can not be prioritized for yggdrasil. + +### infradian.icu and ultradian.club +- `infradian.icu` and `ultradian.club`, run by [Umbrellix](http://yww.umbrellix.net/contact/) + - The SRV record `_xmpp-server._tcp` for all of the relevant subdomains has a domain that points to `202:8fb5:1490:594d:7e29:98fd:8d79:953f` at priority 1, and the clearnet machine which that represents at priority 2 (so that clearnet XMPPers/Snikketers/Jabberers will be able to message you if you are on infradian or ultradian) + - The SRV record `_xmpp-client._tcp` for @ only has the domain which points to `202:8fb5:1490:594d:7e29:98fd:8d79:953f`. There is a non-obvious way to connect over clearnet, which will not be disclosed here. + - HTTP upload, unlike on the other domain Umbrellix has which is clearnet-based, is disabled. ---- From b44049797c059748fd0fc8922280bd434466df6c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?William=20Wennerstr=C3=B6m?= Date: Mon, 9 Mar 2020 22:26:48 +0100 Subject: [PATCH 88/95] Uptime is float as time.Seconds returns a float64 See: https://golang.org/src/time/time.go?s=25778:25813#L792 I stumbled upon this when playing with the Admin API, example: { "peers": { "": { ... "uptime": 625.732186871 ... }, "": { ... "uptime": 625.638570027 ... } } } --- admin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/admin.md b/admin.md index c18536b..cf5caf2 100644 --- a/admin.md +++ b/admin.md @@ -118,7 +118,7 @@ For each IPv6 address: - `bytes_recvd` (`uint64`) contains the number of bytes received from that peer - `endpoint` (`string`) contains the connected IPv4/IPv6 address and port of the peering - `port` (`uint8`) contains the local switch port number for that peer -- `uptime` (`uint32`) contains the number of seconds since the peer connection was established +- `uptime` (`float64`) contains the number of seconds since the peer connection was established #### `addPeer` From 15b2ab0e5038676cc23a6e34a1f16802b9fdb5fa Mon Sep 17 00:00:00 2001 From: Dmitrii <8805785+mflsim@users.noreply.github.com> Date: Thu, 12 Mar 2020 20:51:45 +0300 Subject: [PATCH 89/95] Update services.md --- services.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/services.md b/services.md index e54add5..3af8661 100644 --- a/services.md +++ b/services.md @@ -27,9 +27,12 @@ The following services are available on the Yggdrasil network, courtesy of our c - [Mesh Forum](http://[303:60d4:3d32:a2b9::4]/) - a chit-chat forum about everything. Run by @pztrn. - Accessible via Yggdrasil at `http://[303:60d4:3d32:a2b9::4]/`. -- [Yggedit](http://[301:b614:c68e:b27f::1]/) - a reddit like social network/forum +- [Yggedit](http://[301:b614:c68e:b27f::1]/) - a link aggregator / reddit alternative - Accessible via Yggdrasil at `http://[301:b614:c68e:b27f::1]/`. +- [Yggy Forum](http://[301:b614:c68e:b27f::2]/) - a fast forum with many features + - Accessible via Yggdrasil at `http://[301:b614:c68e:b27f::2]/`. + ### Wikis - [Internal Yggdrasil wiki](http://[203:e0b0:ec08:4e1f:f004:19a9:577a:90ba]/wiki/) run by Jeff at `http://[203:e0b0:ec08:4e1f:f004:19a9:577a:90ba]/wiki/` From eb34d37b12fe7d6e94e611d1f11fe8d43206948f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?William=20Wennerstr=C3=B6m?= Date: Sat, 28 Mar 2020 11:11:37 +0100 Subject: [PATCH 90/95] Installation guide for Gentoo Linux Yggdrasil has been added to the Gentoo GURU repository (somewhat like AUR for Gentoo): https://gitweb.gentoo.org/repo/proj/guru.git/tree/net-p2p/yggdrasil-go This commit adds a guide for installing Yggdrasil using this repository on Gentoo. --- installation-linux-gentoo.md | 79 ++++++++++++++++++++++++++++++++++++ installation.md | 1 + 2 files changed, 80 insertions(+) create mode 100644 installation-linux-gentoo.md diff --git a/installation-linux-gentoo.md b/installation-linux-gentoo.md new file mode 100644 index 0000000..f2462b5 --- /dev/null +++ b/installation-linux-gentoo.md @@ -0,0 +1,79 @@ +--- +tags: dontlink +sitemap: true +--- + +# Installing on Gentoo Linux + +Yggdrasil is hosted in the Gentoo GURU repository as the +[net-p2p/yggdrasil-go](https://gitweb.gentoo.org/repo/proj/guru.git/tree/net-p2p/yggdrasil-go) +package. + +## Package install from the GURU repository + +The GURU repository has to be enabled using `eselect repository`. + +``` +sudo eselect repository enable guru +``` + +And then synchronized with `emerge`: + +``` +sudo emerge --sync +``` + +Now, the `net-p2p/yggdrasil-go` package is masked and marked as unstable using +the `~amd64` keyword. So we have to unmask it by accepting the keyword, there +are two alternatives. + +1. If `/etc/portage/package.accept_keywords` is a directory: + +``` +echo "net-p2p/yggdrasil-go ~amd64" | sudo tee /etc/portage/package.accept_keywords/yggdrasil-go +``` + +2. If `/etc/portage/package.accept_keywords` is a file: + +``` +echo "net-p2p/yggdrasil-go ~amd64" | sudo tee -a /etc/portage/package.accept_keywords +``` + +Now `net-p2p/yggdrasil-go` can be merged and installed with: + +``` +sudo emerge --ask net-p2p/yggdrasil-go +``` + +And you're good to go! + +## Making configuration changes + +Modify the `/etc/yggdrasil.conf` file and then either reload the config: + +1. OpenRC +``` +rc-service yggdrasil reload +``` + +2. systemd +``` +systemctl reload yggdrasil +``` + +... or restart the Yggdrasil daemon altogether: + +1. OpenRC +``` +rc-service yggdrasil restart +``` + +2. systemd +``` +systemctl restart yggdrasil +``` + +## After installation + +Read the [Linux platform page](platform-linux.md) page for further +information about Linux platform support. diff --git a/installation.md b/installation.md index d1abc02..eab7db9 100644 --- a/installation.md +++ b/installation.md @@ -10,6 +10,7 @@ Installation guides are available for the following platforms/distributions. - [Debian, Ubuntu, elementaryOS, Linux Mint and similar](installation-linux-deb.md) - [Red Hat Enterprise Linux, Fedora, CentOS and similar](installation-linux-rpm.md) +- [Gentoo Linux](installation-linux-gentoo.md) - [Ubiquiti EdgeOS](installation-linux-edgeos.md) - [Manual install or build from source](installation-linux-other.md) From 8ecd1344a742ff9574fbb978c03d8a68a6bd784d Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 1 Apr 2020 20:43:16 +0100 Subject: [PATCH 91/95] Update changelog.md --- changelog.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/changelog.md b/changelog.md index ab3e966..6504f9d 100644 --- a/changelog.md +++ b/changelog.md @@ -25,6 +25,14 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - in case of vulnerabilities. --> +## [0.3.14] - 2020-01-01 +### Fixed +- Fixes a memory leak that may occur if packets are incorrectly never removed from a switch queue + +### Changed +- Make DHT searches a bit more reliable by tracking the 16 most recently visited nodes +- CI builds are now built using Go 1.14.1 + ## [0.3.13] - 2020-02-21 ### Added - Support for the Wireguard TUN driver, which now replaces Water and provides far better support and performance on Windows From 9f1c3d6596216cd554cd9c02f4f936ad62b83733 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Wed, 1 Apr 2020 23:22:28 +0100 Subject: [PATCH 92/95] Update index.md --- index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.md b/index.md index d8c06e7..14fbf42 100644 --- a/index.md +++ b/index.md @@ -11,7 +11,7 @@ Although Yggdrasil shares many similarities with [cjdns](https://github.com/cjde ## Project Status -The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.13](changelog.md) and are still regularly working on updates. +The project is currently in early stages but it is being actively developed. We have recently released [version 0.3.14](changelog.md) and are still regularly working on updates. A small number of users have been using and stress-testing Yggdrasil quite heavily for a number of purposes, including but not limited to secure remote access (SSH and VNC), access to Matrix, Jabber and IRC servers and even some video streams, large file transfers and performance tests. We have a number of [internal services](services.md) available, contributed and operated by the community. From f35421354fbbf6cfc5f8b809208a5fc544512900 Mon Sep 17 00:00:00 2001 From: Neil Alexander Date: Sat, 11 Apr 2020 21:57:01 +0100 Subject: [PATCH 93/95] Update services.md --- services.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/services.md b/services.md index e54add5..94a9793 100644 --- a/services.md +++ b/services.md @@ -22,6 +22,11 @@ The following services are available on the Yggdrasil network, courtesy of our c ## Websites +### Search + +- [YaCy](http://[301:4541:2f84:1188:216:3eff:fe38:cefc]:8090/) - an internal search engine. + - Accessible via Yggdrasil at `http://[301:4541:2f84:1188:216:3eff:fe38:cefc]:8090/`. + ### Forums - [Mesh Forum](http://[303:60d4:3d32:a2b9::4]/) - a chit-chat forum about everything. Run by @pztrn. From 1b5687d6b40e14210ba4abfdf0f7428126e08ea0 Mon Sep 17 00:00:00 2001 From: 0bsessive <65446770+0bsessive@users.noreply.github.com> Date: Sat, 16 May 2020 10:32:16 +0000 Subject: [PATCH 94/95] add clearnet tunnel service --- services.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/services.md b/services.md index 94a9793..183356b 100644 --- a/services.md +++ b/services.md @@ -20,6 +20,11 @@ The following services are available on the Yggdrasil network, courtesy of our c ---- +### Clearnet Tunnel Service + +- Automated GRE over Yggdrasil service for clearnet IPv4 access. + - Accessible at http://[301:a003:3f0f:1aa0::1001]/dokuwiki/doku.php?id=ygre:howto + ## Websites ### Search From 2b520b077b60cfa0e734a527ab6d318ac8c047f7 Mon Sep 17 00:00:00 2001 From: George <57254463+zhoreeq@users.noreply.github.com> Date: Sun, 17 May 2020 03:35:26 -0400 Subject: [PATCH 95/95] Underline links --- assets/css/style.scss | 1 + 1 file changed, 1 insertion(+) diff --git a/assets/css/style.scss b/assets/css/style.scss index 01574b1..805ee4b 100644 --- a/assets/css/style.scss +++ b/assets/css/style.scss @@ -159,4 +159,5 @@ div.blogpost div#overview div#excerpt { a:hover, a:focus { color:#0be; font-weight: initial; + text-decoration: underline; }