Failed to establish connection to local host.

[XinY-Z]

I ran into the below error when running NNI in a Conda environment. It has no issue when running in Python virtual environment. To give some background information, I used Python 3.10 in Conda environment (created by Miniconda), installed NNI 3.0 using pip. Previously I was running it on Python 3.6 in venv, and installed NNI 2.6.1 using pip. I got a package that can only run on Python >= 3.8 and that's the only reason why I switched to 3.10. I wonder if anyone ran into similar issue before. I'd appreciate any idea about this error.

`[2023-09-19 00:08:39] Creating experiment, Experiment ID: 7ph1zcg8
[2023-09-19 00:08:39] Starting web server...
/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node: /lib64/libm.so.6: version 'GLIBC_2.27' not found (required by /uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node)
/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node: /lib64/libc.so.6: version 'GLIBC_2.25' not found (required by /uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node)
/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node: /lib64/libc.so.6: version 'GLIBC_2.28' not found (required by /uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node)
/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node: /lib64/libstdc++.so.6: version 'CXXABI_1.3.9' not found (required by /uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node)
/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node: /lib64/libstdc++.so.6: version 'GLIBCXX_3.4.20' not found (required by /uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node)
/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node: /lib64/libstdc++.so.6: version 'GLIBCXX_3.4.21' not found (required by /uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni_node/node)
[2023-09-19 00:08:40] WARNING: Timeout, retry...
[2023-09-19 00:08:41] WARNING: Timeout, retry...
Traceback (most recent call last):
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/connectionpool.py", line 415, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/http/client.py", line 1283, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/http/client.py", line 1329, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x2b779aa92950>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /api/v1/nni/check-status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2b779aa92950>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/PycharmProjects/RoBERTa/tuner.py", line 26, in
experiment.run()
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni/experiment/experiment.py", line 236, in run
return self._run_impl(port, wait_completion, debug)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni/experiment/experiment.py", line 205, in _run_impl
self.start(port, debug)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni/experiment/experiment.py", line 163, in start
self._start_nni_manager(port, debug, run_mode, None, [])
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni/experiment/experiment.py", line 125, in _start_nni_manager
self._proc = launcher.start_experiment(self._action, self.id, config, port, debug, run_mode,
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni/experiment/launcher.py", line 125, in start_experiment
_check_rest_server(port, url_prefix=url_prefix)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni/experiment/launcher.py", line 201, in _check_rest_server
rest.get(port, '/check-status', url_prefix)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni/experiment/rest.py", line 43, in get
return request('get', port, api, prefix=prefix)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/nni/experiment/rest.py", line 31, in request
resp = requests.request(method, url, timeout=timeout)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/uufs/chpc.utah.edu/common/HIPAA/u1318593/software/pkg/miniconda3/envs/RoBERTa/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /api/v1/nni/check-status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2b779aa92950>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-09-19 00:08:42] ERROR: Create experiment failed: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /api/v1/nni/check-status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x2b779aa92950>: Failed to establish a new connection: [Errno 111] Connection refused'))`

[XinY-Z]

Issue has been resolved by downgrading NNI to 2.5.