nextstrain view can't connect to host on a Mac inside a Conda environment

[huddlej]

Current Behavior

As of version 7.3.0.post1, I get the following error when trying to run nextstrain view:

$ nextstrain view auspice                                                                                                                                               [±master ●]
Traceback (most recent call last):
  File "runpy", line 196, in _run_module_as_main
  File "runpy", line 86, in _run_code
  File "nextstrain.cli.__main__", line 55, in <module>
  File "nextstrain.cli.__main__", line 19, in main
  File "nextstrain.cli", line 36, in run
  File "nextstrain.cli.command.view", line 243, in run
  File "nextstrain.cli.command.view", line 400, in resolve
  File "socket", line 955, in getaddrinfo
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

Since the issue occurs since 7.3.0.post and relates to the host or port values, I assumed that it might have originated with #310

My host and port values were:

• $HOST: x86_64-apple-darwin13.4.0
• $PORT is not set

When I tried to provide the same port as the default, I got the same error:

nextstrain view --port 4000 auspice

When I tried to provide the same host as the defaults I did not get the error:

nextstrain view --host localhost auspice

When I tried to provide the same host and port as the defaults, I still did not get the error:

nextstrain view --host localhost --port 4000 auspice

Finally, I tried to provide the same host value as defined in my environment and this produced the error again:

nextstrain view --host "x86_64-apple-darwin13.4.0" auspice

That host name looked unusual, so I checked it and confirmed that it is not a valid host on the network. Searching my environment for host names, I found an odd Conda "backup" host:

$ env | grep -i host
CONDA_BACKUP_HOST=LD7XFMGMG0
HOST=x86_64-apple-darwin13.4.0
CONDA_TOOLCHAIN_HOST=x86_64-apple-darwin13.4.0
host_alias=x86_64-apple-darwin13.4.0

This suggested that Conda has mangled my HOST variable for some reason. To confirm that this is the case, I started a new terminal session and checked host:

$ echo $HOST
LD7XFMGMG0

That value is my actual host name. Then, I activated a Conda environment and checked host:

$ echo $HOST
x86_64-apple-darwin13.4.0

Possible solution

For the short term, my personal solution is either to run nextstrain view outside of a Conda environment or to explicitly pass --host localhost to the command. I could also dig into why Conda changes the HOST variable.

For the long term, I don't know how common this issue is for other Mac/Conda users, but it would be nice to retain the original CLI behavior where localhost was the default. Maybe the CLI could fallback to "localhost" when the environment value of HOST is not valid?

Your environment: if running Nextstrain locally

• Operating system: Mac OS X Monterey 12.6.7
• CLI Version: 7.3.0.post1 and 7.4.0

[tsibley]

We've run into variations of this bug before re: Auspice. I had an inkling of it when writing #310 but thought we were well past it. Guess not! Or I misremembered! orz

The expectation/intent of the code is that HOST is normally not set and thus the default of localhost normally applies. That holds on Linux systems I'm most familiar with. But it seems like maybe its normally set on macOS, even outside of a Conda environment?

Maybe the CLI could fallback to "localhost" when the environment value of HOST is not valid?

This sounds simple but is hardish to do right. And the expectation anyways is that HOST is rarely set and if it is set that it was done so intentionally, and so erroring is correct.

[tsibley]

Relevant Auspice bits that handle this: nextstrain/auspice@8cdbaae

[tsibley]

This sounds simple but is hardish to do right.

Ok, it might be not hard as I was thinking we'd have to proxy errors thru from the runtime, but that's mistaken and we always try to resolve the host outside the runtime anyway, so we can catch it there.

[tsibley]

On a macOS that I have access to, it seems a HOST variable is present in the default interactive shell (zsh) but not exported so it's not part of the environment inherited by child processes. That's not potentially matching the behaviour @huddlej reports above, as it's not clear from examples if HOST is being inherited when outside of a Conda env.

blab@blab-mac ~ % echo $HOST                                              
blab-mac.fhcrc.org
blab@blab-mac ~ % env | grep HOST                             
blab@blab-mac ~ % perl -E 'say "present" if exists $ENV{HOST}'
blab@blab-mac ~ % 

[tsibley]

@huddlej I suspect that inside your Conda env, e.g. at some $prefix, you have a $prefix/etc/conda/activate.d/* script which sets and exports HOST (and also CONDA_BACKUP_HOST).

[tsibley]

Probably from the ctng-compiler-activation and/or clang-compiler-activation packages, which install various activation/deactivation scripts for gcc & co. and clang. Those scripts set HOST and CONDA_BACKUP_HOST, among a sea of other variables, e.g.:

https://github.com/conda-forge/ctng-compiler-activation-feedstock/blob/9661e4c/recipe/activate-gcc.sh#L171-L172

https://github.com/conda-forge/clang-compiler-activation-feedstock/blob/a1d7c95/recipe/activate-clang.sh#L165-L166

Two things I notice about those scripts:

1. They're sourced as opposed to execed (AFAIK), but seemingly fail to differentiate between a shell variable and an environment variable and always export variables when restoring the old/backup value, e.g. CONDA_BACKUP_HOST. IIUC, this means they can end up inadvertently exporting a shell variable into the inherited environment after an activation/deactivation cycle. That's a bug.

2. Since zsh uses the shell variable HOST in its default prompt (via %m), the activation scripts special-case zsh via hook functions to fix the issue they create when they overwrite HOST. e.g. without that workaround, your zsh prompt would look something like huddlej@x86_64-apple-darwin13.4.0 ~ % in the Conda env.

I don't know why those compiler activation packages are getting dragged into your env, probably from some dep, but it's unfortunate when they're used outside of the context of building Conda packages with conda-build (as noted by an open issue).

[tsibley]

# Create Conda env with ctng-compiler-activation package installed.
$ micromamba create -p /tmp/conda -c conda-forge --override-channels gcc_linux-64

# Set HOST as a *shell variable*.
$ HOST=whunk

# It's visible to the shell…
$ echo $HOST
whunk

# …but not part of the inherited environment (i.e. not exported).
$ python3 -c 'import os; print(os.environ.get("HOST"))'
None

# Activate the Conda environment, which sources the ctng-compiler-activation
# scripts, e.g. /tmp/conda/etc/conda/activate.d/activate-gcc.sh.
$ conda activate /tmp/conda

# The activation script sets HOST…
$ echo $HOST
x86_64-conda-linux-gnu

# …and exports it into the inherited environment.
$ python3 -c 'import os; print(os.environ.get("HOST"))'
x86_64-conda-linux-gnu

# Deactivate the Conda environment, which sources the ctng-compiler-activation
# scripts, e.g. /tmp/conda/etc/conda/deactivate.d/deactivate-gcc.sh.
$ conda deactivate

# The deactivation script restores the old HOST value…
$ echo $HOST
whunk

# …but leaves it exported into the inherited environment.
$ python3 -c 'import os; print(os.environ.get("HOST"))'
whunk

[tsibley]

I'm putting this down for now, satisfied that the impact of this issue will be more limited than not, i.e. it's users of Conda envs with particular packages installed, not all Conda envs (I think) and definitely not all macOS users. I wanted to make sure it wasn't more prudent to quickly revert our change, and I don't think it is. We should still address this issue one way or another, and ideally those conda-forge packages would cut it out when it comes to HOST.