- Support for prioritising multiple peerings to the same node has been added, useful for nodes with multiple network interfaces
- The priority can be configured by specifying `?priority=X` in a `Peers` or `Listen` URI, or by specifying `Priority` within a `MulticastInterfaces` configuration entry
- Priorities are values between 0 and 254 (default is 0), lower numbers are prioritised and nodes will automatically negotiate the higher of the two values
### Changed
- On Linux, `SO_REUSEADDR` is now used on the multicast port instead of `SO_REUSEPORT`, which should allow processes running under different users to run simultaneously
### Fixed
- Adding peers using the `InterfacePeers` configuration option should now work correctly again
- Multiple connections from the same remote IP address will no longer be incorrectly dropped
- The admin socket will no longer incorrectly claim TCP connections as TLS
- A panic that could occur when calling `GetPeers` while a peering link is being set up has been fixed
- Support for peering over UNIX sockets is now available, by configuring `Listen` and peering URIs in the `unix:///path/to/socket.sock` format
### Changed
-`yggdrasilctl` has been refactored and now has cleaner output
- It is now possible to `addPeer` and `removePeer` using the admin socket again
- The `getSessions` admin socket call reports number of bytes received and transmitted again
- The link setup code has been refactored, making it easier to support new peering types in the future
- Yggdrasil now maintains configuration internally, rather than relying on a shared and potentially mutable structure
### Fixed
- Tracking information about expired root nodes has been fixed, which should hopefully resolve issues with reparenting and connection failures when the root node disappears
- A bug in the mobile framework code which caused a crash on Android when multicast failed to set up has been fixed
- Yggdrasil should now shut down gracefully and clean up correctly when running as a Windows service
- ICMPv6 "Packet Too Big" payload size has been increased, which should fix Path MTU Discovery (PMTUD) when two nodes have different `IfMTU` values configured
- Previously, addresses were derived from a hash of X25519 (Diffie-Hellman) keys
- Importantly, this means that **all internal IPv6 addresses will change with this release** — this will affect anyone running public services or relying on Yggdrasil for remote access
- There are separate configuration options to control if beacons are sent, what port to listen on for incoming connections (if sending beacons), and whether or not to listen for beacons from other nodes (and open connections when receiving a beacon)
- Each configuration entry in the list specifies a regular expression to match against interface names
- If an interface matches multiple regex in the list, it will use the settings for the first entry in the list that it matches with
- This is still an early work-in-progress, so the code hasn't been as well tested or optimized as the old code base — please bear with us for these next few releases as we work through any bugs or issues
- Generally speaking, we expect to see reduced bandwidth use and improved reliability with the new design, especially in cases where nodes move around or change peerings frequently
- Cryptographic sessions no longer use a single shared (ephemeral) secret for the entire life of the session. Keys are now rotated regularly for ongoing sessions (currently rotated at least once per round trip exchange of traffic, subject to change in future releases)
- Source routing has been added. Under normal circumstances, this is what is used to forward session traffic (e.g. the user's IPv6 traffic)
- DHT-based routing has been added. This is used when the sender does not know a source route to the destination. Forwarding through the DHT is less efficient, but the only information that it requires the sender to know is the destination node's (static) key. This is primarily used during the key exchange at session setup, or as a temporary fallback when a source route fails due to changes in the network
- The new DHT design is no longer RPC-based, does not support crawling and does not inherently allow nodes to look up the owner of an arbitrary key. Responding to lookups is now implemented at the application level and a response is only sent if the destination key matches the node's `/128` IP or `/64` prefix
- The greedy routing scheme, used to forward all traffic in previous releases, is now only used for protocol traffic (i.e. DHT setup and source route discovery)
- The routing logic now lives in a [standalone library](https://github.com/Arceliar/ironwood). You are encouraged **not** to use it, as it's still considered pre-alpha, but it's available for those who want to experiment with the new routing algorithm in other contexts
- Session MTUs may be slightly lower now, in order to accommodate large packet headers if required
- Several remote `debug` functions have been added temporarily, to allow for crawling and census gathering during the transition to the new version, but we intend to remove this at some point in the (possibly distant) future
- The list of available functions will likely be expanded in future releases
- It was far too easy to accidentally break routing altogether by capturing the route to peers with the TUN adapter
- We recommend tunnelling an existing standard over Yggdrasil instead (e.g. `ip6gre`, `ip6gretap` or other similar encapsulations, using Yggdrasil IPv6 addresses as the tunnel endpoints)
- All `TunnelRouting` configuration options will no longer take effect
- This was never a true firewall — it didn't behave like a stateful IP firewall, often allowed return traffic unexpectedly and was simply a way to prevent a node from being flooded with unwanted sessions, so the name could be misleading and usually lead to a false sense of security
- Due to design changes, the new code needs to address the possible memory exhaustion attacks in other ways and a single configurable list no longer makes sense
- Users who want a firewall or other packet filter mechansim should configure something supported by their OS instead (e.g. `ip6tables`)
- All `SessionFirewall` configuration options will no longer take effect
- It was not obvious which parts of the configuration could be reloaded at runtime, and which required the application to be killed and restarted to take effect
- Reloading the config without restarting was also a delicate and bug-prone process, and was distracting from more important developments
-`SIGHUP` will be handled normally (i.e. by exiting)
- Swich lookups happen independently for each (incoming) peer connection, instead of being funneled to a single dedicated switch worker
- Packets are queued for each (outgoing) peer connection, instead of being handled by a single dedicated switch worker
- Queue logic rewritten
- Heap structure per peer that traffic is routed to, with one FIFO queue per traffic flow
- The total size of each heap is configured automatically (we basically queue packets until we think we're blocked on a socket write)
- When adding to a full heap, the oldest packet from the largest queue is dropped
- Packets are popped from the queue in FIFO order (oldest packet from among all queues in the heap) to prevent packet reordering at the session level
- Removed global `sync.Pool` of `[]byte`
- Local `sync.Pool`s are used in the hot loops, but not exported, to avoid memory corruption if libraries are reused by other projects
- This may increase allocations (and slightly reduce speed in CPU-bound benchmarks) when interacting with the tun/tap device, but traffic forwarded at the switch layer should be unaffected
- Support for the Wireguard TUN driver, which now replaces Water and provides far better support and performance on Windows
- Windows `.msi` installer files are now supported (bundling the Wireguard TUN driver)
- NodeInfo code is now actorised, should be more reliable
- The DHT now tries to store the two closest nodes in either direction instead of one, such that if a node goes offline, the replacement is already known
- The Yggdrasil API now supports dialing a remote node using the public key instead of the Node ID
- TAP mode has been removed entirely, since it is no longer supported with the Wireguard TUN package. Please note that if you are using TAP mode, you may need to revise your config!
- NetBSD support has been removed until the Wireguard TUN package supports NetBSD
- Support for TLS listeners and peers has been added, allowing the use of `tls://host:port` in `Peers`, `InterfacePeers` and `Listen` configuration settings - this allows hiding Yggdrasil peerings inside regular TLS connections
- On recent Linux kernels, Yggdrasil will now set the `tcp_congestion_control` algorithm used for its own TCP sockets to [BBR](https://github.com/google/bbr), which reduces latency under load
- The systemd service configuration in `contrib` (and, by extension, some of our packages) now attempts to load the `tun` module, in case TUN/TAP support is available but not loaded, and it restricts Yggdrasil to the `CAP_NET_ADMIN` capability for managing the TUN/TAP adapter, rather than letting it do whatever the (typically `root`) user can do
- Yggdrasil will now complain more verbosely when a peer URI is incorrectly formatted
- Soft-shutdown methods have been added, allowing a node to shut down gracefully when terminated
- New multicast interval logic which sends multicast beacons more often when Yggdrasil is first started to increase the chance of finding nearby nodes quickly after startup
- The switch now buffers packets more eagerly in an attempt to give the best link a chance to send, which appears to reduce packet reordering when crossing aggregate sets of peerings
- Substantial amounts of the codebase have been refactored to use the actor model, which should substantially reduce the chance of deadlocks
- Nonce tracking in sessions has been modified so that memory usage is reduced whilst still only allowing duplicate packets within a small window
- Soft-reconfiguration support has been simplified using new actor functions
- The garbage collector threshold has been adjusted for mobile builds
- The maximum queue size is now managed exclusively by the switch rather than by the core
- Yggdrasil can now send multiple packets from the switch at once, which results in improved throughput with smaller packets or lower MTUs
- Performance has been slightly improved by not allocating cancellations where not necessary
- Crypto-key routing options have been renamed for clarity
-`IPv4Sources` is now named `IPv4LocalSubnets`
-`IPv6Sources` is now named `IPv6LocalSubnets`
-`IPv4Destinations` is now named `IPv4RemoteSubnets`
-`IPv6Destinations` is now named `IPv6RemoteSubnets`
- The old option names will continue to be accepted by the configuration parser for now but may not be indefinitely
- When presented with multiple paths between two nodes, the switch now prefers the most recently used port when possible instead of the least recently used, helping to reduce packet reordering
- New nonce tracking should help to reduce the number of packets dropped as a result of multiple/aggregate paths or congestion control in the switch
- Address verification was not strict enough, which could result in a malicious session sending traffic with unexpected or spoofed source or destination addresses which Yggdrasil could fail to reject
- The switch should now forward packets along a single path more consistently in cases where congestion is low and multiple equal-length paths exist, which should improve stability and result in fewer out-of-order packets
- Sessions should now be more tolerant of out-of-order packets, by replacing a bitmask with a variable sized heap+map structure to track recently received nonces, which should reduce the number of packets dropped due to reordering when multiple paths are used or multiple independent flows are transmitted through the same session
- The admin socket can no longer return a dotfile representation of the known parts of the network, this could be rebuilt by clients using information from `getSwitchPeers`,`getDHT` and `getSessions`
- In low-traffic scenarios where there are multiple peerings between a pair of nodes, Yggdrasil now prefers the most active peering instead of the least active, helping to reduce packet reordering
- The `Listen` statement, when configured as a string rather than an array, will now be parsed correctly
- The admin socket now returns `coords` as a correct array of unsigned 64-bit integers, rather than the internal representation
- Yggdrasil now has a public API with interfaces such as `yggdrasil.ConnDialer`, `yggdrasil.ConnListener` and `yggdrasil.Conn` for using Yggdrasil as a transport directly within applications
- Session gatekeeper functions, part of the API, which can be used to control whether to allow or reject incoming or outgoing sessions dynamically (compared to the previous fixed whitelist/blacklist approach)
- Support for logging to files or syslog (where supported)
- Platform defaults now include the ability to set sane defaults for multicast interfaces
- Multicast discovery is no longer disabled if the nominated interfaces aren't available on the system yet, e.g. during boot
- Multicast interfaces are now re-evaluated more frequently so that Yggdrasil doesn't need to be restarted to use interfaces that have become available since startup
- Admin socket error cases are now handled better
- Various fixes in the TUN/TAP module, particularly surrounding Windows platform support
- Invalid keys will now cause the node to fail to start, rather than starting but silently not working as before
- Session MTUs are now always calculated correctly, in some cases they were incorrectly defaulting to 1280 before
- Multiple searches now don't take place for a single connection
- The `AllowedEncryptionPublicKeys` option has now been fixed to handle incoming connections properly and no longer blocks outgoing connections (this was broken in v0.3.4)
- Multicast TCP listeners will now be stopped correctly when the link-local address on the interface changes or disappears altogether
- On macOS, Yggdrasil will now try to wake up AWDL on start-up when `awdl0` is a configured multicast interface, to keep it awake after system sleep, and to stop waking it when no longer needed
- Added `LinkLocalTCPPort` option for controlling the port number that link-local TCP listeners will listen on by default when setting up `MulticastInterfaces` (a node restart is currently required for changes to `LinkLocalTCPPort` to take effect - it cannot be updated by reloading config during runtime)
- The `Listen` configuration statement is now an array instead of a string
- The `Listen` configuration statement should now conform to the same formatting as peers with the protocol prefix, e.g. `tcp://[::]:0`
- Session workers are now non-blocking
- Multicast interval is now fixed at every 15 seconds and network interfaces are reevaluated for eligibility on each interval (where before the interval depended upon the number of configured multicast interfaces and evaluation only took place at startup)
- Dead connections are now closed in the link handler as opposed to the switch
- Peer forwarding is now prioritised instead of randomised
- Dynamic reconfiguration, which allows reloading the configuration file to make changes during runtime by sending a `SIGHUP` signal (note: this only works with `-useconffile` and not `-useconf` and currently reconfiguring TUN/TAP is not supported)
- Support for building Yggdrasil as an iOS or Android framework if the appropriate tools (e.g. `gomobile`/`gobind` + SDKs) are available
- Connection contexts used for TCP connections which allow more exotic socket options to be set, e.g.
- Reusing the multicast socket to allow multiple running Yggdrasil instances without having to disable multicast
- Allowing supported Macs to peer with other nearby Macs that aren't even on the same Wi-Fi network using AWDL
- Flexible logging support, which allows for logging at different levels of verbosity
- The admin socket is now multithreaded, greatly improving performance of the crawler and allowing concurrent lookups to take place
- The ability to hide NodeInfo defaults through either setting the `NodeInfoPrivacy` option or through setting individual `NodeInfo` attributes to `null`
- DHT entries are now populated using a copy in memory to fix various potential DHT bugs
- DHT traffic should now throttle back exponentially to reduce idle traffic
- Adjust how nodes are inserted into the DHT which should help to reduce some incorrect DHT traffic
- In TAP mode, the NDP target address is now correctly used when populating the peer MAC table. This fixes serious connectivity problems when in TAP mode, particularly on BSD
- In TUN mode, ICMPv6 packets are now ignored whereas they were incorrectly processed before
- The address range was moved from `fd00::/8` to `200::/7`. This range was chosen as it is marked as deprecated. The change prevents overlap with other ULA privately assigned ranges.
- UTF-16 detection conversion for configuration files, which can particularly be a problem on Windows 10 if a configuration file is generated from within PowerShell.
- Use local queue sizes for a sort of local-only backpressure routing, instead of the removed bandwidth estimates, when deciding where to send a packet.
