Hosts can ping lighthouse but not each other ( Handshake timed out )

[kuolemaaa]

Hi everyone
I never managed to make nebula works on my setup and I have no clue on how to "debug" or "trace" a problem on a network aside from ping so here am I desperately writing this discussion after weeks of f**k around.
My last attempt is on 3 machines that are in the same local network 192.168.1.0/24 and I configured nebula for a 192.168.55.0/24 network (unless I did it wrong). As title nebula hosts can ping lighthouse address and vice versa but cannot ping each other.
For simplicity I will rename them alpha beta and omega.

• "Omega", ArchLinux, Nebula 1.8.2 , local_addr: 192.168.1.254, nebula_addr Lighthouse 192.168.55.2
• "Alpha" , Manjaro Linux, Nebula 1.7.2, local_addr: 192.168.1.4, nebula_addr host: 192.168.55.1
• "Beta" , Manjaro Linux, Nebula 1.7.2, local_addr:192.168.1.120, nebula_addr host 192.168.55.6

Notice different nebula version. Could it be a problem?

Alpha nebula config

pki:
  ca: | 
    [REDACTED]
  cert: |
    [REDACTED]
  key: |
    [REDACTED]

static_host_map:
  "192.168.55.2" : ["192.168.1.254:42424"]

lighthouse:
  am_lighthouse: false
  hosts: 
    - "192.168.55.2"
  
punchy:
  punch: true

relay:
  relays:
    - 192.168.55.2


tun:
  disabled: false
  dev: "nebula-network"
  drop_local_broadcast: false
  drop_multicast: false
  mtu: 1300

firewall:
  outbound_action: drop
  inbound_action: drop

  outbound:
    - port: any
      proto: any
      host: any
  
  inbound:
    - port: any
      proto: any
      host: any

Beta nebula config

pki:
  ca: |
    [REDACTED]
  cert: |
    [REDACTED]
  key: |
    [REDACTED]

static_host_map:
  "192.168.55.2" : ["192.168.1.254:42424"]

lighthouse:
  am_lighthouse: false
  hosts: 
    - "192.168.55.2"
  
punchy:
  punch: true

tun:
  disabled: false
  dev: "nebula-network"
  drop_local_broadcast: false
  drop_multicast: false
  mtu: 1300

firewall:
  outbound_action: drop
  inbound_action: drop

  outbound:
    - port: any
      proto: any
      host: any
  
  inbound:
    - port: any
      proto: any
      host: any

Omega nebula config (lighthouse)

pki:
  ca: | 
    [REDACTED]
  cert: |
    [REDACTED]
  key: |
    [REDACTED]


lighthose:
  am_lighthouse: true

listen:
  host: 192.168.1.254
  port: 42424

punchy:
  punch: true


tun:
  disabled: false
  dev: nebula-network
  drop_local_broadcast: false
  drop_multicast: false
  mtu: 1300

logging:
  level: info
  format: text

firewall:
  outbound_action: drop
  inbound_action: drop

  outbound:
    - port: any
      proto: any
      host: any
  
  inbound:
    - port: any
      proto: any
      host: any

An extract of Alpha nebula log, a bunch of:

INFO[3639] Handshake timed out                           durationNs=7162638680 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=158942758 localIndex=158942758 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.6
INFO[3647] Handshake timed out                           durationNs=6755773581 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1667429326 localIndex=1667429326 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.6
INFO[3654] Handshake timed out                           durationNs=6863350656 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1490971935 localIndex=1490971935 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.6
INFO[3661] Handshake timed out                           durationNs=6669932239 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=4291169209 localIndex=4291169209 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.6
INFO[3668] Handshake timed out                           durationNs=6975834099 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2548798329 localIndex=2548798329 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.6

An extract of Beta nebula log, a bunch of:

INFO[3611] Handshake timed out                           durationNs=7171915052 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=129619459 localIndex=129619459 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.1
INFO[3619] Handshake timed out                           durationNs=6962991099 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1053199698 localIndex=1053199698 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.1
INFO[3626] Handshake timed out                           durationNs=6971600662 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=623630284 localIndex=623630284 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.1
INFO[3633] Handshake timed out                           durationNs=7278594714 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2257858750 localIndex=2257858750 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.1
INFO[3642] Handshake timed out                           durationNs=7571626040 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1708889766 localIndex=1708889766 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.1
INFO[3649] Handshake timed out                           durationNs=7164955703 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=3776550701 localIndex=3776550701 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.1
INFO[3657] Handshake timed out                           durationNs=6955881233 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1931752071 localIndex=1931752071 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.1
INFO[3664] Handshake timed out                           durationNs=7063205932 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=332394115 localIndex=332394115 remoteIndex=0 udpAddrs="[]" vpnIp=192.168.55.1

An extract of Omega (lighthouse) nebula log (from journalctl):

level=info msg="Firewall rule added" firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:any ip: localIp: proto:0 startPort:0]"
level=info msg="Firewall rule added" firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: localIp: proto:0 startPort:0]"
level=info msg="Firewall started" firewallHashes="SHA:498215dec4e5687a2353f51c10838c113bd1af35ef72b8e8c9f536986ada5417,FNV:2782948616"
level=info msg="listening \"192.168.1.254\" 42424"
level=info msg="Main HostMap created" network=192.168.55.2/24 preferredRanges="[]"
level=info msg="punchy enabled"
level=warning msg="No lighthouse.hosts configured, this host will only be able to initiate tunnels with static_host_map entries"
level=info msg="Loaded send_recv_error config" sendRecvError=always
level=info msg="Nebula interface is active" boringcrypto=false build=1.8.2 interface=nebula-network network=192.168.55.2/24 udpAddr="192.168.1.254:42424"
Started Nebula overlay networking tool.

level=info msg="Handshake message received" certName=ALPHA fingerprint=[REDACTED] handshake="map[stage:1 style:ix_psk0]" initiatorIndex=3096533120 issuer=[REDACTED] remoteIndex=0 responderIndex=0 udpAddr="192.168.1.4:38121" vpnIp=192.168.55.1
level=info msg="Handshake message sent" certName=ALPHA fingerprint=[REDACTED] handshake="map[stage:2 style:ix_psk0]" initiatorIndex=3096533120 issuer=[REDACTED] remoteIndex=0 responderIndex=2668086302 udpAddr="192.168.1.4:38121" vpnIp=192.168.55.1
level=info msg="Handshake message received" certName=BETA fingerprint=[REDACTED] handshake="map[stage:1 style:ix_psk0]" initiatorIndex=1268016319 issuer=[REDACTED] remoteIndex=0 responderIndex=0 udpAddr="192.168.1.120:47348" vpnIp=192.168.55.6
level=info msg="Handshake message sent" certName=BETA fingerprint=[REDACTED] handshake="map[stage:2 style:ix_psk0]" initiatorIndex=1268016319 issuer=[REDACTED] remoteIndex=0 responderIndex=2718248437 udpAddr="192.168.1.120:47348" vpnIp=192.168.55.6
level=info msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=3887817132 localIndex=3887817132 remoteIndex=0 udpAddrs="[192.168.1.120:47348]" vpnIp=192.168.55.6
level=info msg="Handshake message received" certName=BETA durationNs=812955109 fingerprint=[REDACTED] handshake="map[stage:2 style:ix_psk0]" initiatorIndex=3887817132 issuer=[REDACTED] remoteIndex=3887817132 responderIndex=508333108 sentCachedPackets=1 udpAddr="192.168.1.120:47348" vpnIp=192.168.55.6

level=info msg="Tunnel status" certName=BETA localIndex=2718248437 remoteIndex=1268016319 tunnelCheck="map[method:active state:dead]" vpnIp=192.168.55.6

I'm just noticing that last line on the lighthouse log. I cannot interpret it.

Feel free to ask for more information or more log if you want to help me.

Any help or interest is appreciated, every new notion or tool is precious for me.

Just in case I maybe will regenerate network, certs and configs later.

[kuolemaaa]

update#1: upgraded nebula lightouse on omega (archlinux) to 1.9.0 . still no success.

update#2: got access on nebula's SSH on alpha and trying with another manjaro machine at nebula ip 192.168.55.4/24 (same config of beta) .

On alpha:

nebula@nebula > list-pending-hostmap
192.168.55.4: []
# understandable, that's the problem

nebula@nebula > list-lighthouse-addrmap
192.168.55.2: {"192.168.55.1":{"reported":[{"ip":"192.168.1.254","port":42424}],"relay":[]},"192.168.55.2":{"learned":[{"ip":"192.168.1.254","port":42424}],"relay":[]}}
192.168.55.4: {}
# i'm not sure about that first line and not sure if that's correct.
#     if i am interpreting it correctly it says:
#     lighthouse 192.168.55.2 got 192.168.55.1 & 192.168.55.2
#        192.168.55.1 being reported to 192.168.1.254:42424 ???
#        192.168.55.2 learned from 192.168.1.24:42424 ???
# but 192.168.55.4 is not a Lighthouse. why is it shown here? double checked 192.168.55.4's config

nebula@nebula > query-lighthouse 192.168.55.1 # myself
null
#weird

nebula@nebula > query-lighthouse 192.168.55.4  #the other host
{}
# understandable, that's the problem

nebula@nebula > query-lighthouse 192.168.1.2  # the lighthouse 
null

[johnmaguire]

@kuolemaaa Where you say "understandable" in the output, I actually don't understand where 192.168.55.4 is coming from. In your initial ticket, alpha .2, beta is .6, and omega (lighthouse) is .1 - and indeed, these are the vpnIps that appear in the Lighthouse log file. 😛 Can you help clarify this for me?

Regarding the null result from query-lighthouse, please see the note here and try re-running that command (maybe also with .6, if that's the correct IP address?): https://nebula.defined.net/docs/guides/debug-ssh-commands/#notes-about-some-commands

"reported" refers to the list of IP addresses that were sent to the Lighthouse by a node (i.e. they essentially look at ifconfig and send all known IP addresses to the Lighthouse), while "learned" refers to the src IP address in the handshake packet the Lighthouse received (this is how Lighthouses learn the public IP address of remote nodes - typically nodes won't know it themselves because they are behind a NAT provided by the network router.)

but 192.168.55.4 is not a Lighthouse. why is it shown here? double checked 192.168.55.4's config

list-lighthouse-addrmap... a bit confusingly, internally to Nebula, every node has a "Lighthouse" struct internally that tracks the IP addresses it learned from actual Lighthouse nodes.

[johnmaguire]

Hi @kuolemaaa - for the best chance of NAT hole punching success, please also enable punchy.respond in your hosts' configs.

If they still can't access each other, chances are at least one host (maybe both) is behind a symmetric NAT. To test this, with Go installed, you can run the following command: go run github.com/pion/stun/cmd/stun-nat-behaviour@master 2>&1 | grep "NAT mapping" (see #666 (comment) for details on how to interpret the results.)

In any case, one way to establish connectivity between two nodes that can't seem to talk directly is to create a relay host. You can use your Lighthouse for this - though for larger, more active networks, we recommend using a dedicated host to avoid bogging down the Lighthouse.

[kuolemaaa]

Thanks for the answer.

The fact is that I am not behind NAT here.

I'm trying to make it works on between machines already on my LAN (192.168.1.0/24), needless to say I can reach my machines via normal network.

Changed nebula config on hosts:

punchy:
  punch: true
  respond: true

relay:
  relays:
    - "192.168.55.2" # my lighthouse

changed nebula config on lighthouse:

punchy:
  punch: true
  respond: true

relay:
  am_relay: true

Still no luck. No ping exchanged between hosts.

[johnmaguire]

Hi @kuolemaaa - thanks for clarifying. Unfortunately the snippet of logs you provided only shows handshake messages timing out, and doesn't include the context around them. Therefore, it's difficult to tell what IP addresses the nodes are learning from the Lighthouse or who they are trying to handshake with. Including more logs may offer more insight.

Additionally, you could try taking pcaps on alpha and beta, looking for packets destined for the other.