diff --git a/content/posts/abusing-mesh-networks-for-easy-remote-support.md b/content/posts/abusing-mesh-networks-for-easy-remote-support.md new file mode 100644 index 0000000..f4eabd9 --- /dev/null +++ b/content/posts/abusing-mesh-networks-for-easy-remote-support.md @@ -0,0 +1,563 @@ +--- +title: "(Ab)using mesh networks for easy remote support" +author: ["Amolith"] +date: 2021-11-01T02:51:00-04:00 +lastmod: 2023-01-18T09:33:39-05:00 +tags: ["Mesh networking", "Open source", "Remote support"] +categories: ["Technology"] +draft: false +toc: true +--- + +One of the things many of us struggle with when setting friends and +family up with Linux is remote support. Commercial solutions like +[RealVNC](https://www.realvnc.com/) and [RustDesk](https://rustdesk.com/) do exist and function very well, but are often more +expensive than we would like for answering the odd "I can't get Facebook +open!" support call. I've been on the lookout for suitable alternatives +for a couple years but nothing has been satisfying. Because of this, I +have held off on setting others up with any Linux distribution, even the +particularly user-friendly options such as [Linux Mint](https://linuxmint.com/) and [elementary OS;](https://elementary.io/) +if I'm going drop someone in an unfamiliar environment, I want to be +able to help with any issue within a couple hours, not days and +_certainly_ not weeks. + +[Episode 421 of LINUX Unplugged](https://linuxunplugged.com/421) gave me an awesome idea to use [Nebula,](https://github.com/slackhq/nebula) a +networking tool created by Slack, [X11vnc,](https://libvnc.github.io/) a very minimal VNC server, and +[Remmina,](https://remmina.org/) a libre remote access tool available in pretty much every Linux +distribution, to set up a scalable, secure, and simple setup reminiscent +of products like RealVNC. + + +## Nebula {#nebula} + +The first part of our stack is Nebula, the tool that creates a network +between all of our devices. With traditional VPNs, you have a client +with a persistent connection to a central VPN server and other clients +can communicate with the first by going through that central server. +This works wonderfully in most situations, but there are a lot of +latency and bandwidth restrictions that would make remote support an +unpleasant experience. Instead of this model, what we want is a _mesh_ +network, where each client can connect directly to one another _without_ +going through a central system and slowing things down. This is where +Nebula comes in. + +In Nebula's terminology, clients are referred to as _nodes_ and central +servers are referred to as _lighthouses_, so those are the terms I'll use +going forward. + +Mesh networks are usually only possible when dealing with devices that +have static IP addresses. Each node has to know _how_ to connect with the +other nodes; John can't meet up with Bob when Bob moves every other day +without notifying anyone of his new address. This wouldn't be a problem +if Bob phoned Jill and told her where he was moving; John would call +Jill, Jill would tell him where Bob is, and the two would be able to +find each other + +With Nebula, nodes are Bob and John and Jill is a lighthouse. Each node +connects to a lighthouse and the lighthouse tells the nodes how to +connect with one another when they ask. It _facilitates_ the P2P +connection then _backs out of the way_ so the two nodes can communicate +directly with each other. + +It allows any node to connect with any other node on any network from +anywhere in the world, as long as one lighthouse is accessible that +knows the connection details for both peers. + + +### Getting started {#getting-started} + +The _best_ resource is [the official documentation,](https://github.com/slackhq/nebula) but I'll describe the +process here as well. + +After [installing the required packages,](https://github.com/slackhq/nebula#1-the-nebula-binaries-or-distribution-packages-for-your-specific-platform-specifically-youll-need-nebula-cert-and-the-specific-nebula-binary-for-each-platform-you-use) make sure you have a VPS with a +static IP address to use as a lighthouse. If you want something dirt +cheap, I would recommend one of the small plans from [BuyVM.](https://buyvm.net) I do have a +[referral link](https://my.frantech.ca/aff.php?aff=3783) if you want them to kick me a few dollars for your +purchase. [Hetzner](https://www.hetzner.com/cloud) (referral: `ckGrk4J45WdN`) or [netcup](https://www.netcup.eu/) (referral: +`36nc15758387844`) would also be very good options; I've used them all and +am very comfortable recommending them. + + +### Creating a Certificate Authority {#creating-a-certificate-authority} + +After picking a device with a static IP address, it needs to be set up +as a lighthouse. This is done by first creating a Certificate Authority +(CA) that will be used for signing keys and certificates that allow our +other devices into the network. The `.key` file produced by the following +command is incredibly sensitive; with it, anyone can authorise a new +device and give it access to your network. Store it in a safe, +preferably encrypted location. + +```bash + nebula-cert ca -name "nebula.example.com" +``` + +I'll explain why we used a Fully-Qualified Domain Name (FQDN) as the +CA's name in a later section. If you have your own domain, feel free to +use that instead; it doesn't really matter what domain is used as long +as the format is valid. + + +### Generating lighthouse credentials {#generating-lighthouse-credentials} + +Now that we have the CA's `.crt` and `.key` files, we can create and sign +keys and certificates for the lighthouse. + +```bash + nebula-cert sign -name "buyvm.lh.nebula.example.com" -ip "192.168.100.1/24" +``` + +Here, we're using a FQDN for the same reason as we did in the CA. You +can use whatever naming scheme you like, I just prefer +`.lh.nebula...` for my lighthouses. The IP address can be on any +of the following private IP ranges, I just happened to use `192.168.100.X` +for my network. + +| IP Range | Number of addresses | +|-------------------------------|---------------------| +| 10.0.0.0 – 10.255.255.255 | 16 777 216 | +| 172.16.0.0 – 172.31.255.255 | 10 48 576 | +| 192.168.0.0 – 192.168.255.255 | 65 536 | + + +### Creating a config file {#creating-a-config-file} + +The next step is creating our lighthouse's config file. The reference +config can be found in [Nebula's repo.](https://github.com/slackhq/nebula/blob/master/examples/config.yml) We only need to change a few of +the lines for the lighthouse to work properly. If I don't mention a +specific section here, I've left the default values. + +The section below is where we'll define certificates and keys. `ca.crt` +will remain `ca.crt` when we copy it over but I like to leave the node's +cert and key files named as they were when generated; this makes it easy +to identify nodes by their configs. Once we copy everything over to the +server, we'll add the proper paths to the `cert` and `key` fields. + +```yaml + pki: + ca: /etc/nebula/ca.crt + cert: /etc/nebula/ + key: /etc/nebula/ +``` + +The next section is for identifying and mapping your lighthouses. This +needs to be present in _all_ of the configs on _all_ nodes, otherwise they +won't know how to reach the lighthouses and will never actually join the +network. Make sure you replace `XX.XX.XX.XX` with whatever your VPS's +public IP address is. If you've used a different private network range, +those changes need to be reflected here as well. + +```yaml + static_host_map: + "192.168.100.1": ["XX.XX.XX.XX:4242"] +``` + +Below, we're specifying how the node should behave. It is a lighthouse, +it should answer DNS requests, the DNS server should listen on all +interfaces on port 53, it sends its IP address to lighthouses every 60 +seconds (this option doesn't actually have any effect when `am_lighthouse` +is set to `true` though), and this lighthouse should not send reports to +other lighthouses. The bit about DNS will be discussed later. + +```yaml + lighthouse: + am_lighthouse: true + serve_dns: true + dns: + host: 0.0.0.0 + port: 53 + interval: 60 + hosts: +``` + +The next bit is about [hole punching](https://en.wikipedia.org/wiki/Hole_punching_%28networking%29), also called _NAT punching_, _NAT +busting_, and a few other variations. Make sure you read the comments for +better explanations than I'll give here. `punch: true` enables hole +punching. I also like to enable `respond` just in case nodes are on +particularly troublesome networks; because we're using this as a support +system, we have no idea what networks our nodes will actually be +connected to. We want to make sure devices are available no matter where +they are. + +```yaml + punchy: + punch: true + respond: true + delay: 1s +``` + +`cipher` is a big one. The value _must_ be identical on _all_ nodes _and_ +lighthouses. `chachapoly` is more compatible so it's used by default. The +devices _I_ want to connect to are all x86 Linux, so I can switch to `aes` +and benefit from [a small performance boost.](https://www.reddit.com/r/networking/comments/iksyuu/comment/g3ra5cv/?utm_source=share&utm_medium=web2x&context=3) Unless you know _for sure_ +that you won't need to work with _anything_ else, I recommend leaving it +set to `chachapoly`. + +```yaml + cipher: chachapoly +``` + +The last bit I modify is the firewall section. I leave most everything +default but _remove_ the bits after `port: 443`. I don't _need_ the `laptop` and +`home` groups (groups will be explained later) to access port `443` on this +node, so I shouldn't include the statement. If you have different needs, +take a look at the comment explaining how the firewall portion works and +make those changes. + +Again, I _remove_ the following bit from the config. + +```yaml + - port: 443 + proto: tcp + groups: + - laptop + - home +``` + + +### Setting the lighthouse up {#setting-the-lighthouse-up} + +We've got the config, the certificates, and the keys. Now we're ready to +actually set it up. After SSHing into the server, grab the [latest +release of Nebula for your platform,](https://github.com/slackhq/nebula/releases/latest) unpack it, make the `nebula` binary +executable, then move it to `/usr/local/bin` (or some other location +fitting for your platform). + +```bash + wget https://github.com/slackhq/nebula/releases/download/vX.X.X/nebula-PLATFORM-ARCH.tar.gz + tar -xvf nebula-* + chmod +x nebula + mv nebula /usr/local/bin/ + rm nebula-* +``` + +Now we need a place to store our config file, keys, and certificates. + +```bash + mkdir /etc/nebula/ +``` + +The next step is copying the config, keys, and certificates to the +server. I use `rsync` but you can use whatever you're comfortable with. +The following four files need to be uploaded to the server. + +- `config.yml` +- `ca.crt` +- `buyvm.lh.nebula.example.com.crt` +- `buyvm.lh.nebula.example.com.key` + +With `rsync`, that would look something like this. Make sure `rsync` is also +installed on the VPS before attempting to run the commands though; +you'll get an error otherwise. + +```bash + rsync -avmzz ca.crt user@example.com: + rsync -avmzz config.yml user@example.com: + rsync -avmzz buyvm.lh.* user@example.com: +``` + +SSH back into the server and move everything to `/etc/nebula/`. + +```bash + mv ca.crt /etc/nebula/ + mv config.yml /etc/nebula/ + mv buyvm.lh* /etc/nebula/ +``` + +Edit the config file and ensure the `pki:` section looks something like +this, modified to match your hostnames of course. + +```yaml + pki: + ca: /etc/nebula/ca.crt + cert: /etc/nebula/buyvm.lh.nebula.example.com.crt + key: /etc/nebula/buyvm.lh.nebula.example.com.key +``` + +Run the following command to make sure everything works properly. + +```bash + nebula -config /etc/nebula/config.yml +``` + +The last step is daemonizing Nebula so it runs every time the server +boots. If you're on a machine using systemd, dropping the following +snippet into `/etc/systemd/system/nebula.service` should be sufficient. If +you're using something else, check the [the examples directory](https://github.com/slackhq/nebula/tree/master/examples/) for more +options. + +```text + [Unit] + Description=nebula + Wants=basic.target + After=basic.target network.target + Before=sshd.service + + [Service] + SyslogIdentifier=nebula + ExecReload=/bin/kill -HUP $MAINPID + ExecStart=/usr/local/bin/nebula -config /etc/nebula/config.yml + Restart=always + + [Install] + WantedBy=multi-user.target +``` + +We're almost done! + + +### Setting individual nodes up {#setting-individual-nodes-up} + +This process is almost exactly the same as setting lighthouses up. All +you'll need to do is generate a couple of certs and keys then tweak the +configs a bit. + +The following command creates a new cert/key for USER's node with the IP +address `192.168.100.2`. The resulting files would go on the _remote_ node +not yours. Replace `HOST` and `USER` with fitting values. + +```bash + nebula-cert sign -name "HOST.USER.nebula.example.com" -ip "192.168.100.2/24" +``` + +The following command will create a _similar_ cert/key but it will be part +of the `support` group. The files resulting from this should go on _your_ +nodes. With the config we'll create next, nodes in the `support` group +will be able to VNC and SSH into other nodes. Your nodes need to be in +the `support` group so you'll have access to the others. + +```bash + nebula-cert sign -name "HOST.USER.nebula.example.com" -ip "192.168.100.2/24" -groups "support" +``` + +On to the config now. This tells the node that it is _not_ a lighthouse, +it should _not_ resolve DNS requests, it _should_ ping the lighthouses and +tell them its IP address every 60 seconds, and the node at `192.168.100.1` +is one of the lighthouses it should report to and query from. If you +have more than one lighthouse, add them to the list as well. + +```yaml + lighthouse: + am_lighthouse: false + #serve_dns: false + #dns: + #host: 0.0.0.0 + #port: 53 + interval: 60 + hosts: + - "192.168.100.1" +``` + +The other bit that should be modified is the `firewall:` section and this +is where the groups we created earlier are important. Review its +comments and make sure you understand how it works before proceeding. + +We want to allow inbound connections on ports 5900, the standard port +for VNC, and 22, the standard for SSH. Additionally, we _only_ want to +allow connections from nodes in the `support` group. Any _other_ nodes +should be denied access. + +Note that including this section is not necessary on _your_ nodes, those +in the `support` group. It's only necessary on the remote nodes that +you'll be connecting to. As long as the `outbound:` section in the config +on _your_ node allows any outbound connection, you'll be able to access +other nodes. + +```yaml + - port: 5900 + proto: tcp + groups: + - support + + - port: 22 + proto: tcp + groups: + - support +``` + +The certs, key, config, binary, and systemd service should all be copied +to the same places on all of these nodes as on the lighthouse. + + +## X11vnc {#x11vnc} + +_Alright._ The hardest part is finished. Now on to setting `x11vnc` up on +the nodes you'll be supporting. + +All you should need to do is install `x11vnc` using the package manager +your distro ships with, generate a 20 character password with `pwgen -s +20 1`, run the following command, paste the password, wait for `x11vnc` to +start up, make sure it's running correctly, press `Ctrl` + `C`, then add the +command to the DE's startup applications! + +```bash + x11vnc --loop -usepw -listen -display :0 +``` + +`--loop` tells `x11vnc` to restart once you disconnect from the session. +`-usepw` is pretty self-explanatory. `-listen ` is important; it +tells `x11vnc` to only listen on the node's Nebula IP address. This +prevents randos in a coffee shop from seeing an open VNC port and trying +to brute-force the credentials. `-display :0` just defines which X11 +server display to connect to. + +Some distributions like elementaryOS and those that use KDE and GNOME +will surface a dialogue for managing startup applications if you just +press the Windows (Super) key and type `startup`. If that doesn't work, +you'll have to root around in the settings menus, consult the +distribution's documentation, or ask someone else that might know. + +After adding it to the startup application, log out and back in to make +sure it's running in the background. + + +## Remmina {#remmina} + +Now that our network is functioning properly and the VNC server is set +up, we need something that connects to the VNC server over the fancy +mesh network. Enter [Remmina.](https://remmina.org/) This one goes on _your_ nodes. + +Remmina is a multi-protocol remote access tool available in pretty much +ever distribution's package archive as `remmina`. Install it, launch it, +add a new connection profile in the top left, give the profile a +friendly name (I like to use the name of the person I'll be supporting), +assign it to a group, such as `Family` or `Friends`, set the Protocol to +`Remmina VNC Plugin`, enter the node's Nebula IP address in the Server +field, then enter their username and the 20 character password you +generated earlier. I recommend setting the quality to Poor, but Nebula +is generally performant enough that any of the options are suitable. I +just don't want to have to disconnect and reconnect with a lower quality +if the other person happens to be on a slow network. + +Save and test the connection! + +If all goes well and you see the other device's desktop, you're done +with the VNC section! Now on to SSH. + + +## SSH {#ssh} + +First off, make sure `openssh-server` is installed on the remote node; +`openssh-client` would also be good to have, but from what I can tell, +it's not strictly necessary. You _will_ need `openssh-client` on _your_ node, +however. If you already have an SSH key, copy it over to +`~/.ssh/authorized_keys` on the remote node. If you don't, generate one +with `ssh-keygen -t ed25519`. This will create an Ed25519 SSH key pair. +Ed25519 keys are shorter and faster than RSA and more secure than ECDSA +or DSA. If that means nothing to you, don't worry about it. Just note +than this key might not interact well with older SSH servers; you'll +know if you need to stick with the default RSA. Otherwise, Ed25519 is +the better option. After key generation has finished, copy +`~/.ssh/id_ed25519.pub` (note the `.pub` extension) from your node to +`~/.ssh/authorized_keys` on the remote node. The file _without_ `.pub` is your +_private_ key. Like the Nebula CA certificate we generated earlier, this +is extremely sensitive and should never be shared with anyone else. + +Next is configuring SSH to only listen on Nebula's interface; as with +`x11vnc`, this prevents randos in a coffee shop from seeing an open SSH +port and trying to brute-force their way in. Set the `ListenAddress` +option in `/etc/ssh/sshd_config` to the remote node's Nebula IP address. +If you want to take security a step further, search for +`PasswordAuthentication` and set it to `no`. This means your SSH key is +_required_ for gaining access via SSH. If you mess up Nebula's firewall +rules and accidentally give other Nebula devices access to this machine, +they still won't be able to get in unless they have your SSH key. I +_personally_ recommend disabling password authentication, but it's not +absolutely necessary. After making these changes, run `systemctl restart +sshd` to apply them. + +Now that the SSH server is listening on Nebula's interface, it will +actually fail to start when the machine (re)boots. The SSH server starts +faster than Nebula does, so it will look for the interface before Nebula +has even had a chance to connect. We need to make sure systemd waits for +Nebula to start up and connect before it tells SSH to start; run +`systemctl edit --full sshd` and add the following line in the `[Unit]` +section, above `[Service]`. + +```text + After=nebula.service +``` + +Even now, there's still a bit of a hiccup. Systemd won't start SSH until +Nebula is up and running, which is good. Unfortunately, even after +Nebula has started, it still takes a minute to bring the interface up, +causing SSH to crash. To fix _this_, add the following line directly below +`[Service]`. + +```text + ExecStartPre=/usr/bin/sleep 30 +``` + +If the `sleep` executable is stored in a different location, make sure you +use that path instead. You can check by running `which sleep`. + +When the SSH _service_ starts up, it will now wait an additional 30 +seconds before actually starting the SSH _daemon_. It's a bit of a hacky +solution but it works™. If you come up with something better, please +send it to me and I'll include it in the post! My contact information is +at the bottom of [this site's home page.](/) + +After you've made these changes, run `systemctl daemon-reload` to make +sure systemd picks up on the modified service file, then run `systemctl +restart sshd`. You should be able to connect to the remote node from your +node using the following command. + +```bash + ssh USER@ +``` + +If you want to make the command a little simpler so you don't have to +remember the IP every time, create `~/.ssh/config` on your node and add +these lines to it. + +```text + Host USER + Hostname + User USER +``` + +Now you can just run `ssh USER` to get in. If you duplicate the above +block for all of the remote nodes you need to support, you'll only have +to remember the person's username to SSH into their machine. + + +## Going further with Nebula {#going-further-with-nebula} + +This section explains why we used FQDNs in the certs and why the DNS +resolver is enabled on the lighthouse. + +Nebula ships with a built-in resolver meant specifically for mapping +Nebula node hostnames to their Nebula IP addresses. Running a public DNS +resolver is very much discouraged because it can be abused in terrible +ways. However, the Nebula resolver mitigates this risk because it _only_ +answers queries for Nebula nodes. It doesn't forward requests to any +other servers nor does it attempt to resolve any domain other than what +was defined in its certificate. If you use the example I gave above, +that would be `nebula.example.com`; the lighthouse will attempt to resolve +any subdomain of `nebula.example.com` but it will just ignore `example.com`, +`nebula.duckduckgo.com`, `live.secluded.site`, etc. + +Taking advantage of this resolver requires setting it as your secondary +resolver on any device you want to be able to resolve hostnames from. +If you were to add the lighthouse's IP address as your secondary +resolver on your PC, you could enter `host.user.nebula.example.com` in +Remmina's server settings _instead of_ `192.168.1.2`. + +But how you do so is beyond the scope of this post! + +If you're up for some _more_ shenanigans later on down the line, you could +set up a Pi-Hole instance backed by Unbound and configure Nebula as +Unbound's secondary resolver. With this setup, you'd get DNS-level ad +blocking _and_ the ability to resolve Nebula hostname. Pi-Hole would query +Unbound for `host.user.nebula.example.com`, Unbound would receive no +answer from the root servers because the domain doesn't exist outside of +your VPN, Unbound would fall back to Nebula, Nebula would give it an +answer, Unbound would cache the answer, tell Pi-Hole, Pi-Hole would +cache the answer, tell your device, then your device would cache the +answer, and you can now resolve any Nebula host! + +Exactly how you do _that_ is **_definitely_** beyond the scope of this post :P + +If you set any of this up, I would be interested to hear how it goes! As +stated earlier, my contact information is at the bottom of the site's +home page :)