Pandas and numpy version incompatibilities

[jameshadfield]

Installing into a fresh (python 3.10) environment via pip install -r requirements.txt results in the following error when we import pandas:

Traceback (most recent call last):
  File "/Users/naboo/github/nextstrain/fauna/vdb/avian_flu_upload.py", line 3, in <module>
    import pandas as pd
  File "/Users/naboo/miniconda3/envs/fauna/lib/python3.10/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401
  File "/Users/naboo/miniconda3/envs/fauna/lib/python3.10/site-packages/pandas/compat/__init__.py", line 18, in <module>
    from pandas.compat.numpy import (
  File "/Users/naboo/miniconda3/envs/fauna/lib/python3.10/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module>
    from pandas.util.version import Version
  File "/Users/naboo/miniconda3/envs/fauna/lib/python3.10/site-packages/pandas/util/__init__.py", line 2, in <module>
    from pandas.util._decorators import (  # noqa:F401
  File "/Users/naboo/miniconda3/envs/fauna/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly
  File "/Users/naboo/miniconda3/envs/fauna/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Looks like this is due to:

The reason is that pandas defines its numpy dependency freely as "anything newer than certain version of numpy". The problem occured, when numpy==2.0.0 has been released on June 16th 2024, because it is no longer compatible with your pandas version. The solution is to pin down the numpy version to any before the 2.0.0.

Indeed, pip install numpy==1.26.4 fixes this bug.

I suggest we add this pin to requirements.txt. I'm happy to do so, but it's been a long time since I was in the fauna ecosystem so is this the right solution?

[jameshadfield]

Similarly, subsequent dependency mismatch error:

ImportError: Pandas requires version '2.0.1' or newer of 'xlrd' (version '1.2.0' currently installed).

The salient parts of our requirements.txt are:

fauna/requirements.txt

Lines 4 to 8 in d632af0

	pandas >=1.1.5, ==1.*
	rethinkdb >=2.4.8, ==2.4.*, !=2.4.10
	requests >=2.20.0, ==2.*
	unidecode >=1.0.22, ==1.*
	xlrd >=1.0.0, ==1.*

(which brought in pandas 1.5.3 in my case)

Solution: pip install 'xlrd>=2.0.1'

[joverlee521]

I suggest we add this pin to requirements.txt. I'm happy to do so, but it's been a long time since I was in the fauna ecosystem so is this the right solution?

Yeah, I think we want to pin to the earlier numpy version. There's a whole dependency mess discussed in #169.

[jameshadfield]

Solution: pip install 'xlrd>=2.0.1'

I just noticed

fauna/README.md

Line 32 in d632af0

> **_note_**: A subset of tdb upload scripts use `xlrd` (v1.2.0) which will not work with Python 3.8 or newer

So maybe we need to pin pandas low-enough that it can use xlrd 1.2.0 (I don't need the tbd upload scripts so I'm not exploring this at the moment)