264 lines
11 KiB
Markdown
264 lines
11 KiB
Markdown
# Heimdall
|
|
Heimdall is a peer-to-peer private network system. It uses [Wireguard](https://www.wireguard.com) for peer
|
|
communication and [Redis](https://redis.io) for backend storage. All nodes are equal and the system
|
|
auto-elects in the event of a current master failure. Non-node peers are also supported in which they
|
|
can only access the network but do not provide routing capabilities.
|
|
|
|
|
|
|
|
# Concepts
|
|
The following is the core design of Heimdall.
|
|
|
|
## Backend
|
|
[Redis](https://redis.io) is used to store network state. It operates in a master/replica design for
|
|
fault tolerance and high availability.
|
|
|
|
## Security
|
|
Security and privacy are a core component of the network. UUIDs are generated for nodes and peers
|
|
and both can use optional TLS to connect to the network. All communication happens over Wireguard or minimal
|
|
management over the GRPC API.
|
|
|
|
For connectivity, you will need to configure firewalls to allow the GRPC port (default 9000) and the Endpoint
|
|
Port (Wireguard, default: 10100).
|
|
|
|
## Node
|
|
A Node is a machine in the network that operates as a gateway. Nodes get a /16 by default to provide
|
|
network access to services. Since all nodes are created equal, a pre-shared cluster key is used for access
|
|
when joining. Upon joining, the node's Redis store is configured as a replica of the current master.
|
|
If the master goes away, an "oldest sibling" master election is performed. The new master node's Redis
|
|
store is re-configured as the master and all other peer nodes are re-configured as replicas.
|
|
|
|
## Peer
|
|
There is also the ability for non-node peers to join. These peers can access all services provided by the
|
|
gateway nodes but cannot provide routing or access themselves. They are access only peers. In order for
|
|
a peer to join, their peer ID must be authorized by an existing node.
|
|
|
|
## Routes
|
|
In the event that the node's /16 network space is not enough or wants to provide access to another subnet,
|
|
custom routes can be published. This is done by publishing the route via the desired node ID. All nodes
|
|
and peers will sync and re-configure their route tables accordingly.
|
|
|
|
# Setup
|
|
The following is a quick start to get Heimdall running. This has only been tested on Alpine linux but
|
|
should be similar with other distros.
|
|
|
|
## Dependencies
|
|
The following need to be installed for Heimdall to operate properly:
|
|
|
|
Note: the `edge` repositories must be enabled to get the Wireguard packages. Uncomment them from `/etc/apk/repositories`.
|
|
|
|
- `wireguard-vanilla` or `wireguard-virt` if using a VM
|
|
- `wireguard-tools`
|
|
- `iptables`
|
|
- `ip6tables`
|
|
- `redis` (optional)
|
|
|
|
This can be peformed with the following:
|
|
|
|
```bash
|
|
$> apk add -U wireguard-vanilla wireguard-tools iptables ip6tables redis
|
|
```
|
|
|
|
Note: make sure to reboot after installation so the kernel modules can be properly loaded.
|
|
|
|
Enable IP forwarding:
|
|
|
|
```bash
|
|
$> cat << EOF > /etc/sysctl.conf
|
|
net.ipv4.ip_forward = 1
|
|
EOF
|
|
```
|
|
|
|
# Redis
|
|
It is recommended to run a local Redis datastore on the node as all data is replicated.
|
|
It must be configured to bind (`bind 0.0.0.0`) on all interfaces so it can be accessed over the Wireguard
|
|
tunnel once Heimdall creates it. Due to [Protected Mode](https://redis.io/topics/security)
|
|
in Redis it is recommended to use a password via the `requirepass` directive.
|
|
|
|
If using Alpine, you will also want to ensure Redis starts on boot:
|
|
|
|
```bash
|
|
$> rc-update add redis default
|
|
$> service redis start
|
|
```
|
|
|
|
# Heimdall
|
|
Use the following to get the latest version of Heimdall.
|
|
|
|
## Build
|
|
Assuming you have a [Go](https://golang.org) toolchain configured you should be able to run `make` to get binaries.
|
|
|
|
## Releases
|
|
See the releases page for the latest pre-built binaries.
|
|
|
|
# Running
|
|
To start a network use the following. In this example, we will be using a three node network:
|
|
|
|
On the first node, run Heimdall:
|
|
|
|
```bash
|
|
$> heimdall -D -r redis://root:<password>@127.0.0.1:6379 --cluster-key 05e73f5c-2848-434c-a0d8-ce4bb78bd4d3
|
|
DEBU[0000] starting grpc server addr="tcp://192.168.122.198:9000"
|
|
INFO[0000] starting heimdall commit=f845269a.m version=0.1.0
|
|
INFO[0000] starting as master id=09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0000] starting master heartbeat
|
|
DEBU[0000] disabling replica status
|
|
DEBU[0000] starting master heartbeat: ttl=5s
|
|
INFO[0000] cluster master key=05e73f5c-2848-434c-a0d8-ce4bb78bd4d3
|
|
WARN[0000] node does not have an IP assigned yet
|
|
DEBU[0000] generating new keypair for 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0000] allocated network 10.11.0.0/16 for 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0000] updating node network: id=09ed6b23-c608-5863-9b4b-f6d8b75c85ef subnet=10.11.0.0/16
|
|
DEBU[0000] peer info updated: id=09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0000] updating peer config to version 6cf910f3d2aeb5424f6568092014d2a826a269230dad729db93834b03f2df4ca
|
|
DEBU[0000] starting node heartbeat: ttl=15s
|
|
INFO[0000] restarting tunnel darknet
|
|
DEBU[0000] starting peer config updater: ttl=10s
|
|
```
|
|
|
|
Note: if no cluster-key is specified one will be generated. This will need to be specified on future invocations as it is random.
|
|
|
|
On the first node, it allocated `10.11.0.0/16` as its network. It will have the IP `10.11.0.1/16` assigned.
|
|
|
|
On the second node, run Heimdall specifying the first node as the peer:
|
|
|
|
```bash
|
|
$> heimdall -D --cluster-key 05e73f5c-2848-434c-a0d8-ce4bb78bd4d3 -r redis://root:<password>@127.0.0.1:6379 --peer tcp://192.168.122.198:9000
|
|
DEBU[0000] starting grpc server addr="tcp://192.168.122.143:9000"
|
|
INFO[0000] starting heimdall commit=f845269a.m version=0.1.0
|
|
DEBU[0000] joining tcp://192.168.122.198:9000
|
|
DEBU[0000] master info received: id=09ed6b23-c608-5863-9b4b-f6d8b75c85ef grpc=tcp://192.168.122.198:9000
|
|
DEBU[0000] updating peer config to version 1577951667592b0851e6fbfa53a5c8076aeda1266f9855df06d1da081c34a8b7
|
|
INFO[0000] restarting tunnel darknet
|
|
INFO[0000] waiting for master 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0005] rtt master ping: 5.232848429s
|
|
DEBU[0005] joining master: 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
INFO[0005] configuring node as replica of 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0005] configuring redis as slave of 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
INFO[0005] waiting for redis sync with 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0006] starting replica monitor: ttl=5s
|
|
DEBU[0006] node network for f20c02a1-6fd1-5a8d-9df1-caeccdd987c5: 10.12.0.0/16
|
|
DEBU[0006] starting node heartbeat: ttl=15s
|
|
DEBU[0006] starting peer config updater: ttl=10s
|
|
```
|
|
|
|
On the second node, it allocated `10.12.0.0/16` as its network. It will have the IP `10.12.0.1/16` assigned.
|
|
|
|
On the third node, run Heimdall specifying either one of the other nodes as the peer (any node can be joined as it forwards the request to the master):
|
|
|
|
```bash
|
|
$> heimdall -D --cluster-key 05e73f5c-2848-434c-a0d8-ce4bb78bd4d3 --peer tcp://192.168.122.143:9000 -r redis://root:darknet@127.0.0.1:6379
|
|
DEBU[0000] starting grpc server addr="tcp://192.168.122.109:9000"
|
|
INFO[0000] starting heimdall commit=f845269a.m version=0.1.0
|
|
DEBU[0000] joining tcp://192.168.122.143:9000
|
|
DEBU[0000] master info received: id=09ed6b23-c608-5863-9b4b-f6d8b75c85ef grpc=tcp://192.168.122.198:9000
|
|
DEBU[0000] updating peer config to version 477833e21934c0f4dfb3cb8a2786d61444038a057e18e9c041c44fbe47f2d7a0
|
|
INFO[0000] restarting tunnel darknet
|
|
INFO[0000] waiting for master 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0010] rtt master ping: 10.369636565s
|
|
DEBU[0010] joining master: 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
INFO[0010] configuring node as replica of 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0010] configuring redis as slave of 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
INFO[0010] waiting for redis sync with 09ed6b23-c608-5863-9b4b-f6d8b75c85ef
|
|
DEBU[0011] starting replica monitor: ttl=5s
|
|
DEBU[0011] node network for 9e915536-830e-555b-b74c-d4495c0c3467: 10.13.0.0/16
|
|
DEBU[0011] starting node heartbeat: ttl=15s
|
|
DEBU[0011] starting peer config updater: ttl=10s
|
|
```
|
|
|
|
On the third node, it allocated `10.13.0.0/16` as its network. It will have the IP `10.13.0.1/16` assigned.
|
|
|
|
You can view the status of the Wireguard tunnel (the peers will vary depending on which node you check the status) as well as verify connectivity:
|
|
|
|
```bash
|
|
$> wg
|
|
interface: darknet
|
|
public key: yOA4f4iyxFg6spDUGy6uS3M12d5/vkQMfdLgCK27mwI=
|
|
private key: (hidden)
|
|
listening port: 10100
|
|
|
|
peer: Lbcue7KFOqDE2rN1Or2f5hSxE7+ruJJjzOOSpk2m23s=
|
|
endpoint: 192.168.122.143:10100
|
|
allowed ips: 10.12.0.0/16
|
|
latest handshake: 1 minute, 56 seconds ago
|
|
transfer: 36.70 KiB received, 43.65 KiB sent
|
|
|
|
peer: /2YDhxBQQQbP5D+Yu0v07QvBwEnOBNZ6aFtwikhn4C4=
|
|
endpoint: 192.168.122.109:10100
|
|
allowed ips: 10.13.0.0/16
|
|
latest handshake: 1 minute, 56 seconds ago
|
|
transfer: 39.08 KiB received, 47.66 KiB sent
|
|
```
|
|
|
|
From the first node you should be able to ping the other nodes:
|
|
|
|
```bash
|
|
$> ip a s darknet
|
|
55: darknet: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
|
|
link/none
|
|
inet 10.11.0.1/16 scope global darknet
|
|
valid_lft forever preferred_lft forever
|
|
|
|
$> ping 10.12.0.1
|
|
PING 10.12.0.1 (10.12.0.1): 56 data bytes
|
|
64 bytes from 10.12.0.1: seq=0 ttl=64 time=1.643 ms
|
|
64 bytes from 10.12.0.1: seq=1 ttl=64 time=1.757 ms
|
|
64 bytes from 10.12.0.1: seq=2 ttl=64 time=0.429 ms
|
|
|
|
$> ping 10.13.0.1
|
|
PING 10.13.0.1 (10.13.0.1): 56 data bytes
|
|
64 bytes from 10.13.0.1: seq=0 ttl=64 time=1.743 ms
|
|
64 bytes from 10.13.0.1: seq=1 ttl=64 time=1.705 ms
|
|
64 bytes from 10.13.0.1: seq=2 ttl=64 time=0.992 ms
|
|
```
|
|
|
|
# Peers
|
|
To start a non-node peer using the following:
|
|
|
|
First, get the ID of the peer:
|
|
|
|
```bash
|
|
$> hpeer -v
|
|
peer version=0.1.0-dev (f845269a.m) linux/amd64 id=9e915536-830e-555b-b74c-d4495c0c3467
|
|
```
|
|
|
|
Next, authorize the ID in the cluster:
|
|
|
|
```bash
|
|
$> hctl -a tcp://192.168.122.198:9000 peers authorize 9e915536-830e-555b-b74c-d4495c0c3467
|
|
```
|
|
|
|
Verify the ID was authorized:
|
|
|
|
```bash
|
|
$> hctl -a tcp://192.168.122.198:9000 peers authorized
|
|
ID
|
|
9e915536-830e-555b-b74c-d4495c0c3467
|
|
```
|
|
|
|
Now connect the peer:
|
|
|
|
```bash
|
|
$> hpeer -D --addr tcp://192.168.122.198:9000
|
|
INFO[0000] starting heimdall commit=f845269a.m version=0.1.0
|
|
INFO[0000] connecting to peer tcp://192.168.122.198:9000
|
|
DEBU[0000] updating peer config to version 5055b697213dd024f5f001169a1c1f5558739603299ceaa0ff1e17ced15ee1e9
|
|
INFO[0000] restarting tunnel darknet
|
|
```
|
|
|
|
The peer will be allocated an IP on the peer network (default: `10.51.0.0/16`).
|
|
You can now verify connectivity:
|
|
|
|
```bash
|
|
$> ip a s darknet
|
|
5: darknet: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
|
|
link/none
|
|
inet 10.51.0.2/32 scope global darknet
|
|
valid_lft forever preferred_lft forever
|
|
|
|
$> ping 10.11.0.1
|
|
PING 10.11.0.1 (10.11.0.1): 56 data bytes
|
|
64 bytes from 10.11.0.1: seq=0 ttl=64 time=4.157 ms
|
|
64 bytes from 10.11.0.1: seq=1 ttl=64 time=1.582 ms
|
|
64 bytes from 10.11.0.1: seq=2 ttl=64 time=1.177 ms
|
|
```
|