diff --git a/_posts/2021-06-26-v0-4-prerelease-benchmarks.md b/_posts/2021-06-26-v0-4-prerelease-benchmarks.md new file mode 100644 index 0000000..a95a216 --- /dev/null +++ b/_posts/2021-06-26-v0-4-prerelease-benchmarks.md @@ -0,0 +1,56 @@ +--- +layout: post +title: "v0.4 Pre-release Benchmarks" +date: 2021-06-26 21:00:00 +0000 +author: Arceliar +--- + +### Improvements in v0.4 + +As noted in a [recent post](2021-06-19-preparing-for-v0-4.md), the upcoming v0.4 release will include a number of major changes to how Yggdrasil routes traffic. +Most of these changes aim to improve performance in dynamic networks and reduce bandwidth consumption from protocol traffic. +It will take some time to get a sense for how these affect performance in a live network, but until then, I thought it could be interesting to look at some benchmark results. + +### Mesh Network Lab + +All of the results shown here are from [meshnet-lab](https://github.com/mwarning/meshnet-lab). Meshnet-lab simulates mesh networks using linux's network namespace functionality. Each node is give a network namespace, which can be linked to other namespaces to simulate an arbitrary topology. + +Although meshnet-lab supports many other mesh networking protocols, I thought it would be best to focus on comparing Yggdrasil `v0.3.16` (the latest stable release) with `v0.4rc3` (the most recent release candidate). The point of this post is to highlight what kind of performance changes we expect to see in the new Yggdrasil release, not to compare Yggdrasil to other mesh routers. + +#### Mobility1 + +If I understand correctly, the `mobility1` benchmark simulates a dynamic unit disc graph. A two-dimensinal plane of nodes is simulated, with nodes having connections to other nodes that fall within a certain radius. The network periodically moves all nodes a random distance between 0 and X (X=10,30,60m) in a 1km x 1km virtual space, then waits some amount of time (10s or 30s) before pinging 200 random paths. The paths are limited to source/destination pairs that are in the same connected component, so it only tests paths that plausibly could work. + +![mobility1-10-10_arrival_progress](/assets/images/2021-06-26/mobility1-10-10_arrival_progress.svg) +![mobility1-10-30_arrival_progress](/assets/images/2021-06-26/mobility1-10-30_arrival_progress.svg) +![mobility1-10-60_arrival_progress](/assets/images/2021-06-26/mobility1-10-60_arrival_progress.svg) + +![mobility1-30-10_arrival_progress](/assets/images/2021-06-26/mobility1-30-10_arrival_progress.svg) +![mobility1-30-30_arrival_progress](/assets/images/2021-06-26/mobility1-30-30_arrival_progress.svg) +![mobility1-30-60_arrival_progress](/assets/images/2021-06-26/mobility1-30-60_arrival_progress.svg) + +These mobility tests are an area where Yggdrasil has struggled up to now, as seen in the `v0.3.16` results. Basically, when a node moves, this can affect the coords of other nodes in the network. With the changes in `v0.4rc3`, the 30s tests are generally not a problem. The 10s tests see some loss, due to the time it takes to detect failed links before we can route around them. + +#### Mobility2 + +The `mobility2` test is essentially a variation of the above. Nodes periodically move a random (increasing) step size with a 15s delay before testing 200 random paths. This test also monitors bandwidth usage. + +![mobility2_arrival_progress](/assets/images/2021-06-26/mobility2_arrival_progress.svg) +![mobility2_traffic_progress](/assets/images/2021-06-26/mobility2_traffic_progress.svg) + +The main feature to note is that, asside from having terrible reliability in this test, `v0.3.16` uses a ridiculous amount of bandwidth when mobility is involved. With `v0.4rc3`, the bandwith use drops to at or below around 10KBps, depending on how mobile things are. I'm fairly certain that most of this bandwith is still a reaction to mobility events in the network, because (as we're about to see) the bandwith use a pretty low in static networks. + +#### Scalability1 + +The `scalability1` test set involves running the network over line, tree, or square grid networks. The line and tree networks start at 50 nodes and increase to 300. The grid network starts at 49 nodes (7x7) and increases the side length by 1 at each step, up to 298 nodes (17x17). This test waits for about 5 minutes before pinging 200 paths (slowly, over an additional 5 minutes), and measures both packet delivery rate and network utilization. + +![scalability1-line](/assets/images/2021-06-26/scalability1-line.svg) +![scalability1-rtree](/assets/images/2021-06-26/scalability1-rtree.svg) +![scalability1-grid](/assets/images/2021-06-26/scalability1-grid.svg) + +There's not a whole lot to say here, `v0.4rc3` is just an improvement across the board. Note that it's a little surprising how the bandwidth use *decreases* as the network grows. I think this is an artifact of how the test works. Each network measures reliability by pinging a fixed number of paths (200). The bandwidth used by these pings counts towards the test results. In the line network, increasing the network size also increases the path length at an equal rate, so the bandwidth use per node stays about the same. In the grid and rtree networks, path length doesn't increases as rapidly as the number of nodes, so the bandwith from the 200 test pings is increasing slower than the network size, which results in decreased average bandwidth use per node. In the future, it may be interesting to run some variation of this test without the pings, to get a better measurement of how the idle protocol traffic scales. + +### Conclusion + +The upcoming v0.4 release changes how packets are routed through the network. It's hard to predict exactly how this will affect network performance, but benchmarks in simulated networks may give us some insight into what we can expect. If the above benchmarks are at least qualitatively accurate, then we have reason to be optimistic about performance in the next release. + diff --git a/assets/images/2021-06-26/mobility1-10-10_arrival_progress.svg b/assets/images/2021-06-26/mobility1-10-10_arrival_progress.svg new file mode 100644 index 0000000..f395742 --- /dev/null +++ b/assets/images/2021-06-26/mobility1-10-10_arrival_progress.svg @@ -0,0 +1,337 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 60 + + + + + + + + + + + + + 80 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 5 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 15 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 25 + + + + + + + + + + + + + 30 + + + + + + + + + packet arrival [%] + + + + + 10s steps [-] + + + + + Mobility1 test of 50 nodes. Start inside 1x1km square. + + + Step duration is 10 seconds. Step width is 0-10m. 100MBit/s - 1ms latency links. + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/mobility1-10-30_arrival_progress.svg b/assets/images/2021-06-26/mobility1-10-30_arrival_progress.svg new file mode 100644 index 0000000..31a90d9 --- /dev/null +++ b/assets/images/2021-06-26/mobility1-10-30_arrival_progress.svg @@ -0,0 +1,337 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 60 + + + + + + + + + + + + + 80 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 5 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 15 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 25 + + + + + + + + + + + + + 30 + + + + + + + + + packet arrival [%] + + + + + 10s steps [-] + + + + + Mobility1 test of 50 nodes. Start inside 1x1km square. + + + Step duration is 10 seconds. Step width is 0-30m. 100MBit/s - 1ms latency links. + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/mobility1-10-60_arrival_progress.svg b/assets/images/2021-06-26/mobility1-10-60_arrival_progress.svg new file mode 100644 index 0000000..ad17ee8 --- /dev/null +++ b/assets/images/2021-06-26/mobility1-10-60_arrival_progress.svg @@ -0,0 +1,337 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 60 + + + + + + + + + + + + + 80 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 5 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 15 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 25 + + + + + + + + + + + + + 30 + + + + + + + + + packet arrival [%] + + + + + 10s steps [-] + + + + + Mobility1 test of 50 nodes. Start inside 1x1km square. + + + Step duration is 10 seconds. Step width is 0-60m. 100MBit/s - 1ms latency links. + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/mobility1-30-10_arrival_progress.svg b/assets/images/2021-06-26/mobility1-30-10_arrival_progress.svg new file mode 100644 index 0000000..e273259 --- /dev/null +++ b/assets/images/2021-06-26/mobility1-30-10_arrival_progress.svg @@ -0,0 +1,337 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 60 + + + + + + + + + + + + + 80 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 5 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 15 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 25 + + + + + + + + + + + + + 30 + + + + + + + + + packet arrival [%] + + + + + 30s steps [-] + + + + + Mobility1 test of 50 nodes. Start inside 1x1km square. + + + Step duration is 30 seconds. Step width is 0-10m. 100MBit/s - 1ms latency links. + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/mobility1-30-30_arrival_progress.svg b/assets/images/2021-06-26/mobility1-30-30_arrival_progress.svg new file mode 100644 index 0000000..9e6e841 --- /dev/null +++ b/assets/images/2021-06-26/mobility1-30-30_arrival_progress.svg @@ -0,0 +1,337 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 60 + + + + + + + + + + + + + 80 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 5 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 15 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 25 + + + + + + + + + + + + + 30 + + + + + + + + + packet arrival [%] + + + + + 30s steps [-] + + + + + Mobility1 test of 50 nodes. Start inside 1x1km square. + + + Step duration is 30 seconds. Step width is 0-30m. 100MBit/s - 1ms latency links. + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/mobility1-30-60_arrival_progress.svg b/assets/images/2021-06-26/mobility1-30-60_arrival_progress.svg new file mode 100644 index 0000000..880f3cc --- /dev/null +++ b/assets/images/2021-06-26/mobility1-30-60_arrival_progress.svg @@ -0,0 +1,337 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 60 + + + + + + + + + + + + + 80 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 5 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 15 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 25 + + + + + + + + + + + + + 30 + + + + + + + + + packet arrival [%] + + + + + 30s steps [-] + + + + + Mobility1 test of 50 nodes. Start inside 1x1km square. + + + Step duration is 30 seconds. Step width is 0-60m. 100MBit/s - 1ms latency links. + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/mobility2_arrival_progress.svg b/assets/images/2021-06-26/mobility2_arrival_progress.svg new file mode 100644 index 0000000..25f68f8 --- /dev/null +++ b/assets/images/2021-06-26/mobility2_arrival_progress.svg @@ -0,0 +1,430 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 60 + + + + + + + + + + + + + 80 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 5 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 15 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 25 + + + + + + + + + + + + + 30 + + + + + + + + + + + + + 35 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 45 + + + + + + + + + + + + + 50 + + + + + + + + + packet arrival [%] + + + + + Mobility2 Test for 50 randomly placed nodes in a 1x1km square. + + + Move in random directions of 50-400m in 50m increments. + + + Wait and measure ping arrival over 60s in 10s intervals each time. + + + 100MBit/s - 1ms latency links. + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/mobility2_traffic_progress.svg b/assets/images/2021-06-26/mobility2_traffic_progress.svg new file mode 100644 index 0000000..381cf04 --- /dev/null +++ b/assets/images/2021-06-26/mobility2_traffic_progress.svg @@ -0,0 +1,443 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 200 + + + + + + + + + + + + + 300 + + + + + + + + + + + + + 400 + + + + + + + + + + + + + 500 + + + + + + + + + + + + + 600 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 5 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 15 + + + + + + + + + + + + + 20 + + + + + + + + + + + + + 25 + + + + + + + + + + + + + 30 + + + + + + + + + + + + + 35 + + + + + + + + + + + + + 40 + + + + + + + + + + + + + 45 + + + + + + + + + + + + + 50 + + + + + + + + + tx traffic per node [KB/s] + + + + + Mobility2 Test for 50 randomly placed nodes in a 1x1km square. + + + Move in random directions of 50-400m in 50m increments. + + + Wait and measure ping arrival over 60s in 10s intervals each time. + + + 100MBit/s - 1ms latency links. + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/scalability1-grid4.svg b/assets/images/2021-06-26/scalability1-grid4.svg new file mode 100644 index 0000000..e2d8689 --- /dev/null +++ b/assets/images/2021-06-26/scalability1-grid4.svg @@ -0,0 +1,408 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 0.5 + + + + + + + + + + + + + 1 + + + + + + + + + + + + + 1.5 + + + + + + + + + + + + + 2 + + + + + + + + + + + + + 2.5 + + + + + + + + + + + + + 3 + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 50 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 150 + + + + + + + + + + + + + 200 + + + + + + + + + + + + + 250 + + + + + + + + + + + + + 300 + + + + + 0 + + + + + 10 + + + + + 20 + + + + + 30 + + + + + 40 + + + + + 50 + + + + + 60 + + + + + 70 + + + + + 80 + + + + + 90 + + + + + 100 + + + + + + + + + tx per node [KB/s] + + + + + packet arrival [%] + + + + + # number of nodes + + + + + Traffic by routing protocol on grid4 dataset with 100MBit/s - 1ms latency links. + + + 1. Start daemons, 2. Wait 300s, 3. Measure for 300s with <node_count> random pings + + + + + v0.3.16 [KB/s/node] + + + v0.3.16 [KB/s/node] + + + + + + + + + + + + + + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + + + + + + v0.4rc3 [KB/s/node] + + + v0.4rc3 [KB/s/node] + + + + + + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/scalability1-line.svg b/assets/images/2021-06-26/scalability1-line.svg new file mode 100644 index 0000000..a7dcc46 --- /dev/null +++ b/assets/images/2021-06-26/scalability1-line.svg @@ -0,0 +1,414 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 2 + + + + + + + + + + + + + 4 + + + + + + + + + + + + + 6 + + + + + + + + + + + + + 8 + + + + + + + + + + + + + 10 + + + + + + + + + + + + + 12 + + + + + + + + + + + + + 14 + + + + + + + + + + + + + 16 + + + + + + + + + + + + + 18 + + + + + + + + + + + + + 50 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 150 + + + + + + + + + + + + + 200 + + + + + + + + + + + + + 250 + + + + + + + + + + + + + 300 + + + + + 0 + + + + + 10 + + + + + 20 + + + + + 30 + + + + + 40 + + + + + 50 + + + + + 60 + + + + + 70 + + + + + 80 + + + + + 90 + + + + + 100 + + + + + + + + + tx per node [KB/s] + + + + + packet arrival [%] + + + + + # number of nodes + + + + + Traffic by routing protocol on line dataset with 100MBit/s - 1ms latency links. + + + 1. Start daemons, 2. Wait 300s, 3. Measure for 300s with <node_count> random pings + + + + + v0.3.16 [KB/s/node] + + + v0.3.16 [KB/s/node] + + + + + + + + + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + v0.4rc3 [KB/s/node] + + + v0.4rc3 [KB/s/node] + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/images/2021-06-26/scalability1-rtree.svg b/assets/images/2021-06-26/scalability1-rtree.svg new file mode 100644 index 0000000..8831639 --- /dev/null +++ b/assets/images/2021-06-26/scalability1-rtree.svg @@ -0,0 +1,375 @@ + + + +Gnuplot +Produced by GNUPLOT 5.2 patchlevel 8 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + 0.2 + + + + + + + + + + + + + 0.4 + + + + + + + + + + + + + 0.6 + + + + + + + + + + + + + 0.8 + + + + + + + + + + + + + 1 + + + + + + + + + + + + + 1.2 + + + + + + + + + + + + + 50 + + + + + + + + + + + + + 100 + + + + + + + + + + + + + 150 + + + + + + + + + + + + + 200 + + + + + + + + + + + + + 250 + + + + + + + + + + + + + 300 + + + + + 0 + + + + + 10 + + + + + 20 + + + + + 30 + + + + + 40 + + + + + 50 + + + + + 60 + + + + + 70 + + + + + 80 + + + + + 90 + + + + + 100 + + + + + + + + + tx per node [KB/s] + + + + + packet arrival [%] + + + + + # number of nodes + + + + + Traffic by routing protocol on rtree dataset with 100MBit/s - 1ms latency links. + + + 1. Start daemons, 2. Wait 300s, 3. Measure for 300s with <node_count> random pings + + + + + v0.3.16 [KB/s/node] + + + v0.3.16 [KB/s/node] + + + + + + + + + + + + + v0.3.16 [%] + + + v0.3.16 [%] + + + + + + + + + + + + + v0.4rc3 [KB/s/node] + + + v0.4rc3 [KB/s/node] + + + + + + + + + + + + + v0.4rc3 [%] + + + v0.4rc3 [%] + + + + + + + + + + + + + + + + + + + + + + + + +