Public lighthouses

[magg00]

There was a question in an issue about public lighthouses:

#642

I was curious about the security implications of this and any ways to mitigate them. Firewall rules could be used along with groups to isolate each ip, but would there still be a risk of ddos or other threats? What about if each was also given it’s own subnet with /32? I’m trying to understand the different implications.

[solarkraft]

From #642:

Nebula OSS has no plans to run a public Lighthouse - this is left as an exercise to the users

This is totally legitimate, of course, though I do think that this would greatly benefit the project, as it would make it much easier to get into.

Multiple people (such as me) seem to be interested in running public lighthouses, but are unsure about how to do so. Support in this area would be great as I really think such a community would be highly beneficial to the Nebula project as a whole.

[johnmaguire]

I can understand home users' desire for a public Lighthouse. Home users' deployment of Nebula can be painful since it requires a server with a publicly routable fixed IP address and port which usually translates to paying for a cloud server. The problem is that Nebula was really not designed with this use case (public, shared Lighthouses) in mind.

Nebula was built at Slack to solve specific pain points of deploying a large, private network. As such it assumes that one organization owns a CA and issue certificates in a valid manner. This means you need a central authority to track assigned IP addresses to reduce collisions and even more importantly you must be able to trust the person who controls the CA private key, because they can - at any time - add hosts to the network with any name, IP address, and groups. These are all the identifying pieces of information used in firewall rules.

I think that in order to run a public Lighthouse you would really want it to be multi-tenant-aware which would greatly increase the complexity of the source code and is not currently a design goal of Nebula.

It may be possible to do today, but it would be very convoluted and it's not recommended or supported it. Thinking through it quickly, here's how I would start to think about it:

• Create a CA used only to sign the lighthouse. You probably want to assign it an IP address in the CGNAT space (e.g. 100.64.0.1/10) so that you have lots of addressable space for your clients.
• Run the lighthouse with tun disabled and no inbound firewall rules.
• Create an upload form for users to upload their own CA certificate, and add it to the trusted CAs in the lighthouse. This step has some gotchas, because will need each user to create a CA which uses the -ips flag to scope it down to (a) a subnet which falls within the CGNAT address space, and (b) is limited to a smaller size so that a single user can't exhaust the CGNAT space. If the user can upload to this form over and over, they can still quickly exhaust the space. But there's an even bigger issue: The -ips flag restricts which IP space the CA can sign nodes for, which also restricts which IP space a node can talk to. Since you don't want them to be able to sign a node certificate matching the Lighthouse IP address, the node also won't be able to talk to the Lighthouse you created in step 1. I don't know that there's a solution in Nebula for this today.
• The user now needs to download the lighthouse's CA and add it to their trusted CAs list in their config, along with their own CA.
• The user can then create hosts using their own CA.
• The user's firewall rules must include the ca_sha targeting their CA SHA for every single rule in order to ensure that the public Lighthouse operator can't mint their own hosts which are able to access your boxes.

Hopefully this helps to illustrate some of the pitfalls you might run into. The primary one is that I don't think Nebula supports this in a secure way today, period. Even if it did, you will need to ensure that your users don't exhaust your IP address space. Finally, it will be very easy for users to unintentionally allow your Lighthouse access to their machines.

[magg00]

Just to clarify, my main question here is about security and sandboxing the ip addresses rather than a Nebula public instance. I’m aware of the IP clash issue, but I’m hoping to get some thoughts from the devs on the subnet isolation idea and any security implications I may have overlooked.

[solarkraft]

Ha, sorry, I jumped on it with my general remark because of the title.

[SamSirry]

If you limit users to subnets, eventually we will run out of IPs.
Then another additional identifier will need to be used (e.g. a group name, or a group number), and this will need to be included in all packets, which requires a redesign of the protocol, followed up by security testing, etc..

IMO this is not an easy job at all.

[magg00]

I was thinking of having the light house subnet cover /24 but then each individual device inside that subnet cover /32 (just itself). It would mean the devices couldn’t talk to each other but all could talk to the lighthouse? It’s a unique use case and trying to foresee security implications or whether there are risks to one device from another owned by another person.

[bloominstrong]

nebula-lighthouse-service gets around a few of the issues mentioned by:

• Each user controls their own CA.
• Each user gets a separate nebula instance listening on it's own port, which isolates the each users network and gets rid of duplicated IP ranges.
• The public lighthouse sets tun: disabled:true so the interface is not created on the lighthouse, this means that the lighthouse cannot be reached inside the nebula network.
• The user is instructed to put all nodes but the public lighthouse in a group and add that group to all firewall rules groups: trusted_node

Logs on the public lighthouse will show all certificate names and IP addresses which might be a concern.
And packets will only be sent though nebula if the IP range if it is in the certificate or defined as an unsafe range so you cannot try to put the lighthouse on some other subnet so using /32 networks are not going to work.

[maggie44]

I was hoping the subnet specification would limit users so that a device can only see a lighthouse and not other devices:

./nebula-cert sign -name 'test' -ip '192.168.100.10/32' -subnets '192.168.86.0/32','192.168.100.10/32'
./nebula-cert print -path test.crt                                                                    
NebulaCertificate {
        Details {
                Name: test
                Ips: [
                        192.168.100.10/32
                ]
                Subnets: [
                        192.168.86.0/32
                        192.168.100.10/32
                ]
                Groups: []
                Not before: 2023-09-10 15:48:47 +0200 SAST
                Not After: 2024-09-09 15:45:25 +0200 SAST
                Is CA: false
                Issuer: 6a5e56ff07b3cf621e3f2a6ce1e13450964f755357d5f489c43fa24261a208e8
                Public key: 8c302538e124a19dac49c3a768f067937a6c94e26931054701dc38aab8fab904
                Curve: CURVE25519
        }
        Fingerprint: 0c9a98e4f89af1b84935a975d28d0d9777258af93a7e96408ee4ba701e8e702e
        Signature: 11a7941d7587433e8a0b023f4d0d5303aab7e9e9b03f43ce5da4cbab55fbf159f995042b7e6d04c19cb758286dd9319392ed5e4100170f985a79d71e6d866d0a
}

Haven't tested any variations of these, just thinking out loud to see if anyone has any insight.

In my case, I am trying to just connect devices to a lighthouse, not to connect devices to each other. Just one connection (node -> lighthouse) as a private connection.

[bloominstrong]

So there's a few things there,

• a /32 network will not allow you to talk to the lighthouse (or any other node) on the nebula IP you would need to use a /30 so you could have test @ 192.168.86.10/30 and the lighthouse @ 192.168.86.11/30 and 192.168.100.10/30 and then have test2 @ 192.168.100.11/30 , this does mean the lighthouse would end up with many IP addresses and recreating the cert for every new node.
• This idea of everything going through the lighthouse is pretty much going back to traditional VPN model of all nodes routing traffic through a central server, in which case you are still needing to trust the server is not reading or altering the data as it goes through. The way nebula works is the lighthouse orchestras direct connections between nodes.
• If you use the built in sshServer you can explicitly tell nebula to connect with create-tunnel not sure if that would work for nodes with different IP ranges but if they could then a nodes could be able to send traffic directly to another node.

create-tunnel - Creates a tunnel for the provided vpn ip and address
  The lighthouses will be queried for real addresses but you can provide one as well.
  -address string
    	Optionally provide a real remote address, ip:port 

In my case, I am trying to just connect devices to a lighthouse, not to connect devices to each other. Just one connection (node -> lighthouse) as a private connection.

If this is your actual use case Wireguard would be a better option.

The tricky thing is you want three things (in default configuration) to be able to use a public lighthouse,

1. the lighthouse should only advertise your nodes to your nodes and not advertise your nodes to other nodes.
2. other nodes should not be able to handshake with your nodes
3. the lighthouse should not be able to reach your nodes inside the nebula network.

I don't think 1 is possible with nebula unless you run one instance per group of nodes. 2 doable multiple instances but is easy to get wrong with one instance and ca_sha firewall rules and multiple CAs as mentioned above. A good light house will use tun: disabled:true ut a bad one won't so you are limited to group: firewall rules.
While I think nebula-lighthouse-service is secure enough it is not with a "default" config.

[maggie44]

How about each user having their own CA but the lighthouse having multiple CA’s? Is this what you were referring to on the public lighthouse, and the separate instance is just for additional but not necessary security (and presumably to prevent IP overlap)?

[bloominstrong]

How about each user having their own CA but the lighthouse having multiple CA’s? Is this what you were referring to on the public lighthouse

The lighthouse does need to have a CA that the node also has so they can connect, so you could have that plus the users own CA but the issue is the lighthouse CA could create other certificates under it's CA which all the user's nodes trust because they have that CA. The only way to stop that is to add that firewall rule (ca_sha) at each node so they would be able to connect but the firewall would stop connections inside the nebula network.

and the separate instance is just for additional but not necessary security (and presumably to prevent IP overlap)?

It provides security by separating the lighthouse service for each user on the lighthouse on different ports, this means an evil user would need to breakout of their nebula process on the lighthouse to try to attack you from your nebula process on the lighthouse. I would argue that the necessary security would be secure by default, meaning that if the user makes no configuration changes other that the ones required to connect they should be secure, this is not the case in either approach unfortunately. Other people might have a different opinion on what is necessary though.

[maggie44]

Thanks to some help from the Nebula guys on the slack channel, here is an outline of how to create clusters of devices in their own private network all from the same lighthouse, but without ability to access each other.

Note that this doesn't resolve all the issues. There can still be IP clashes, and this doesn't solve all the issues for a potential public lighthouse, but it is a process for certificate sandboxing that could be part of the puzzle and provides one way to isolate groups of devices in a tamper proof way:

Generate CAs:

./nebula-cert ca -name "LH-1" -out-crt "LH-1.crt" -out-key "LH-1.key"
./nebula-cert ca -name "CA-1" -out-crt "CA-1.crt" -out-key "CA-1.key"
./nebula-cert ca -name "CA-2" -out-crt "CA-2.crt" -out-key "CA-2.key"

Lighthouse:

./nebula-cert sign -name "lighthouse" -ip "192.168.100.1/24" -ca-crt "LH-1.crt" -ca-key "LH-1.key"

Devices:

Device1:

./nebula-cert sign -name "device1-CA-1" -ip "192.168.100.2/24" -ca-crt "CA-1.crt" -ca-key "CA-1.key"

Device2:

./nebula-cert sign -name "device2-CA-2" -ip "192.168.100.3/24" -ca-crt "CA-2.crt" -ca-key "CA-2.key"

Lighthouse example:

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    LH-1.crt here
    -----END NEBULA CERTIFICATE-----
    -----BEGIN NEBULA CERTIFICATE-----
    CA-1.crt here
    -----END NEBULA CERTIFICATE-----
    -----BEGIN NEBULA CERTIFICATE-----
    CA-2.crt here
    -----END NEBULA CERTIFICATE-----
  cert: |
    -----BEGIN NEBULA CERTIFICATE-----
   lighthouse.crt here
    -----END NEBULA CERTIFICATE-----

Device 1 example:

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    LH-1.crt here
    -----END NEBULA CERTIFICATE-----
    -----BEGIN NEBULA CERTIFICATE-----
    CA-1.crt here
    -----END NEBULA CERTIFICATE-----
  cert: |
    -----BEGIN NEBULA CERTIFICATE-----
    device1-CA-1.crt here
    -----END NEBULA CERTIFICATE-----

Device 2 example:

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    LH-1.crt here
    -----END NEBULA CERTIFICATE-----
    -----BEGIN NEBULA CERTIFICATE-----
    CA-2.crt here
    -----END NEBULA CERTIFICATE-----
  cert: |
    -----BEGIN NEBULA CERTIFICATE-----
    device2-CA-2.crt here
    -----END NEBULA CERTIFICATE-----