Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support python-311-minimal container image #642

Merged
merged 3 commits into from
Oct 25, 2023

Conversation

phracek
Copy link
Member

@phracek phracek commented Oct 10, 2023

This pull request adds support for python-311-minimal container image
for Fedora, RHEL8 and CentOS Stream 9.

@phracek
Copy link
Member Author

phracek commented Oct 10, 2023

Let's try first round of tests

[test]

@phracek phracek force-pushed the support_python311_minimal branch from a3b8602 to bbb2585 Compare October 10, 2023 13:09
@phracek
Copy link
Member Author

phracek commented Oct 10, 2023

[test]

@frenzymadness
Copy link
Member

  • numpy-test-app application image failed to build with a strange error grep: warning: stray \ before -.
  • 3.11-minimal/test/from-dockerfile/uwsgi.Dockerfile.tpl is just a symlink now but it has to be generated for each Python version because it contains hardcoded 3.9 now.
  • Also, readme is full of references to 3.9 and has to be modified into a template I guess.

@phracek
Copy link
Member Author

phracek commented Oct 17, 2023

I have got error trace from numpy:

Ignoring gunicorn: markers 'python_version < "3.5"' don't match your environment
Ignoring numpy: markers 'python_version < "3.6"' don't match your environment
Collecting gunicorn>=20.0.0
  Downloading gunicorn-21.2.0-py3-none-any.whl (80 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.2/80.2 kB 6.9 MB/s eta 0:00:00
Collecting numpy<1.23.0
  Downloading numpy-1.22.4.zip (11.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.5/11.5 MB 113.2 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting packaging
  Downloading packaging-23.2-py3-none-any.whl (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.0/53.0 kB 311.5 MB/s eta 0:00:00
Building wheels for collected packages: numpy
  Building wheel for numpy (pyproject.toml): started
  Building wheel for numpy (pyproject.toml): finished with status 'error'
Failed to build numpy
  error: subprocess-exited-with-error

  × Building wheel for numpy (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [277 lines of output]
      setup.py:66: RuntimeWarning: NumPy 1.22.4 may not yet support Python 3.11.
        warnings.warn(
      Running from numpy source directory.
      Cythonizing sources
      Processing numpy/random/_bounded_integers.pxd.in
      Processing numpy/random/_mt19937.pyx
      Processing numpy/random/_generator.pyx
      Processing numpy/random/_philox.pyx
      Processing numpy/random/_bounded_integers.pyx.in
      Processing numpy/random/_common.pyx
      Processing numpy/random/_sfc64.pyx
      Processing numpy/random/bit_generator.pyx
      Processing numpy/random/_pcg64.pyx
      Processing numpy/random/mtrand.pyx
      INFO: blas_opt_info:
      INFO: blas_armpl_info:
      INFO: customize UnixCCompiler
      INFO:   libraries armpl_lp64_mp not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: blas_mkl_info:
      INFO:   libraries mkl_rt not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: blis_info:
      INFO:   libraries blis not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: openblas_info:
      INFO:   libraries openblas not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: accelerate_info:
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_3_10_blas_threads_info:
      INFO: Setting PTATLAS=ATLAS
      INFO:   libraries tatlas not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64/sse2', '/usr/lib64', '/usr/lib/sse2', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_3_10_blas_info:
      INFO:   libraries satlas not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64/sse2', '/usr/lib64', '/usr/lib/sse2', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_blas_threads_info:
      INFO: Setting PTATLAS=ATLAS
      INFO:   libraries ptf77blas,ptcblas,atlas not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64/sse2', '/usr/lib64', '/usr/lib/sse2', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_blas_info:
      INFO:   libraries f77blas,cblas,atlas not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64/sse2', '/usr/lib64', '/usr/lib/sse2', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      /tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/system_info.py:2077: UserWarning:
          Optimized (vendor) Blas libraries are not found.
          Falls back to netlib Blas library which has worse performance.
          A better performance should be easily gained by switching
          Blas library.
        if self._calc_info(blas):
      INFO: blas_info:
      INFO:   libraries blas not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      /tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/system_info.py:2077: UserWarning:
          Blas (http://www.netlib.org/blas/) libraries not found.
          Directories to search for the libraries can be specified in the
          numpy/distutils/site.cfg file (section [blas]) or by setting
          the BLAS environment variable.
        if self._calc_info(blas):
      INFO: blas_src_info:
      INFO:   NOT AVAILABLE
      INFO:
      /tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/system_info.py:2077: UserWarning:
          Blas (http://www.netlib.org/blas/) sources not found.
          Directories to search for the sources can be specified in the
          numpy/distutils/site.cfg file (section [blas_src]) or by setting
          the BLAS_SRC environment variable.
        if self._calc_info(blas):
      INFO:   NOT AVAILABLE
      INFO:
      non-existing path in 'numpy/distutils': 'site.cfg'
      INFO: lapack_opt_info:
      INFO: lapack_armpl_info:
      INFO:   libraries armpl_lp64_mp not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: lapack_mkl_info:
      INFO:   libraries mkl_rt not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: openblas_lapack_info:
      INFO:   libraries openblas not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: openblas_clapack_info:
      INFO:   libraries openblas,lapack not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: flame_info:
      INFO:   libraries flame not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_3_10_threads_info:
      INFO: Setting PTATLAS=ATLAS
      INFO:   libraries tatlas,tatlas not found in /opt/app-root/lib
      INFO:   libraries tatlas,tatlas not found in /usr/local/lib64
      INFO:   libraries tatlas,tatlas not found in /usr/local/lib
      INFO:   libraries tatlas,tatlas not found in /usr/lib64/sse2
      INFO:   libraries tatlas,tatlas not found in /usr/lib64
      INFO:   libraries tatlas,tatlas not found in /usr/lib/sse2
      INFO:   libraries tatlas,tatlas not found in /usr/lib
      INFO: <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_3_10_info:
      INFO:   libraries satlas,satlas not found in /opt/app-root/lib
      INFO:   libraries satlas,satlas not found in /usr/local/lib64
      INFO:   libraries satlas,satlas not found in /usr/local/lib
      INFO:   libraries satlas,satlas not found in /usr/lib64/sse2
      INFO:   libraries satlas,satlas not found in /usr/lib64
      INFO:   libraries satlas,satlas not found in /usr/lib/sse2
      INFO:   libraries satlas,satlas not found in /usr/lib
      INFO: <class 'numpy.distutils.system_info.atlas_3_10_info'>
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_threads_info:
      INFO: Setting PTATLAS=ATLAS
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /opt/app-root/lib
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/lib64/sse2
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/lib64
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
      INFO:   libraries ptf77blas,ptcblas,atlas not found in /usr/lib
      INFO: <class 'numpy.distutils.system_info.atlas_threads_info'>
      INFO:   NOT AVAILABLE
      INFO:
      INFO: atlas_info:
      INFO:   libraries f77blas,cblas,atlas not found in /opt/app-root/lib
      INFO:   libraries f77blas,cblas,atlas not found in /usr/local/lib64
      INFO:   libraries f77blas,cblas,atlas not found in /usr/local/lib
      INFO:   libraries f77blas,cblas,atlas not found in /usr/lib64/sse2
      INFO:   libraries f77blas,cblas,atlas not found in /usr/lib64
      INFO:   libraries f77blas,cblas,atlas not found in /usr/lib/sse2
      INFO:   libraries f77blas,cblas,atlas not found in /usr/lib
      INFO: <class 'numpy.distutils.system_info.atlas_info'>
      INFO:   NOT AVAILABLE
      INFO:
      INFO: lapack_info:
      INFO:   libraries lapack not found in ['/opt/app-root/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
      INFO:   NOT AVAILABLE
      INFO:
      /tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/system_info.py:1902: UserWarning:
          Lapack (http://www.netlib.org/lapack/) libraries not found.
          Directories to search for the libraries can be specified in the
          numpy/distutils/site.cfg file (section [lapack]) or by setting
          the LAPACK environment variable.
        return getattr(self, '_calc_info_{}'.format(name))()
      INFO: lapack_src_info:
      INFO:   NOT AVAILABLE
      INFO:
      /tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/system_info.py:1902: UserWarning:
          Lapack (http://www.netlib.org/lapack/) sources not found.
          Directories to search for the sources can be specified in the
          numpy/distutils/site.cfg file (section [lapack_src]) or by setting
          the LAPACK_SRC environment variable.
        return getattr(self, '_calc_info_{}'.format(name))()
      INFO:   NOT AVAILABLE
      INFO:
      INFO: numpy_linalg_lapack_lite:
      INFO:   FOUND:
      INFO:     language = c
      INFO:     define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]
      INFO:
      Warning: attempted relative import with no known parent package
      /tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'define_macros'
        warnings.warn(msg)
      running bdist_wheel
      running build
      running config_cc
      INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
      running config_fc
      INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
      running build_src
      INFO: build_src
      INFO: building py_modules sources
      creating build
      creating build/src.linux-x86_64-3.11
      creating build/src.linux-x86_64-3.11/numpy
      creating build/src.linux-x86_64-3.11/numpy/distutils
      INFO: building library "npymath" sources
      WARN: Could not locate executable armflang
      WARN: Could not locate executable gfortran
      WARN: Could not locate executable f95
      WARN: Could not locate executable ifort
      WARN: Could not locate executable ifc
      WARN: Could not locate executable lf95
      WARN: Could not locate executable pgfortran
      WARN: Could not locate executable nvfortran
      WARN: Could not locate executable f90
      WARN: Could not locate executable f77
      WARN: Could not locate executable fort
      WARN: Could not locate executable efort
      WARN: Could not locate executable efc
      WARN: Could not locate executable g77
      WARN: Could not locate executable g95
      WARN: Could not locate executable pathf95
      WARN: Could not locate executable nagfor
      WARN: Could not locate executable frt
      WARN: don't know how to compile Fortran code on platform 'posix'


      [Errno 2] No such file or directory: 'gcc'


      Traceback (most recent call last):
        File "/opt/app-root/lib64/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/opt/app-root/lib64/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 230, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 268, in run_setup
          self).run_setup(setup_script=setup_script)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 461, in <module>
          setup_package()
        File "setup.py", line 453, in setup_package
          setup(**metadata)
        File "/tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/core.py", line 169, in setup
          return old_setup(**new_attr)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/command/build.py", line 61, in run
          old_build.run(self)
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-4ekbna70/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/command/build_src.py", line 144, in run
          self.build_sources()
        File "/tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/command/build_src.py", line 155, in build_sources
          self.build_library_sources(*libname_info)
        File "/tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/command/build_src.py", line 288, in build_library_sources
          sources = self.generate_sources(sources, (lib_name, build_info))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/distutils/command/build_src.py", line 378, in generate_sources
          source = func(extension, build_dir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-install-rlvllaql/numpy_6d3e78432eb541a38cd4fa85a2a92cb9/numpy/core/setup.py", line 760, in get_mathlib_info
          raise RuntimeError(
      RuntimeError: Broken toolchain: cannot link a simple C program.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for numpy
ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects

[notice] A new release of pip available: 22.2.2 -> 23.3
[notice] To update, run: pip install --upgrade pip
Error: building at STEP "RUN /usr/libexec/s2i/assemble": while running runtime: exit status 1

@frenzymadness
Copy link
Member

I see where the problem lies. For some reason, we are installing numpy<1.23.0 but that's kinda old (1.22.4 released in May 2022) and that version does not provide pre-compiled wheels for Python 3.11 so pip tries to compile it from sources during installation but that fails because gcc and many more dependencies are missing. Is it possible to release that requirement in test or is it from somewhere else?

@phracek
Copy link
Member Author

phracek commented Oct 23, 2023

I see where the problem lies. For some reason, we are installing numpy<1.23.0 but that's kinda old (1.22.4 released in May 2022) and that version does not provide pre-compiled wheels for Python 3.11 so pip tries to compile it from sources during installation but that fails because gcc and many more dependencies are missing. Is it possible to release that requirement in test or is it from somewhere else?

It is hard-coded here: https://github.com/sclorg/s2i-python-container/blob/master/examples/numpy-test-app/requirements.txt#L7.
My idea would be to move the code to dist-gen generated sources and modify them to old pythons, like 3.9 and lower and another code for python 3.11 and higher. Like it is specified here: https://github.com/sclorg/s2i-python-container/tree/master/src/test/from-dockerfile or by similar way.

@phracek
Copy link
Member Author

phracek commented Oct 24, 2023

numpy-test-app is fixed by 92da730

@phracek
Copy link
Member Author

phracek commented Oct 24, 2023

[test]

@frenzymadness
Copy link
Member

Thank you very much for working on this! You did a great job.

There is only one thing I'd change (simplify). The requirements file in numpy-test-app does not have to be generated by distgen because the environment markers can handle the different Python versions like this:

gunicorn<20.0.0; python_version < '3.5'
gunicorn>=20.0.0; python_version >= '3.5'

# Numpy 1.17 dropped support for Python 2
numpy<1.17.0; python_version < '3.6'

# Numpy 1.23.0 no longer builds from source
# on RHEL 7 with old GCC
numpy<1.23.0; python_version >= "3.6" and python_version < "3.11"

# numpy 1.23.5 is the first one with wheels for Python 3.11
numpy>1.23.5; python_version >= "3.11"

I tested the exact string above in multiple virtual environments with various Python versions and it works like a charm. So you can just copy-paste it. And there is one big advantage - we don't have to update it when we add a new Python version. We can keep using this and installing the latest greatest numpy until it breaks with Python 3.11 and then we just limit the version for the latest stable and add a new line for newer Pythons.

The container will be available for Fedora, C9S and RHEL8

Signed-off-by: Petr "Stone" Hracek <[email protected]>
Signed-off-by: Petr "Stone" Hracek <[email protected]>
@phracek phracek force-pushed the support_python311_minimal branch from 9df3777 to 7fbe30d Compare October 24, 2023 15:39
@phracek
Copy link
Member Author

phracek commented Oct 24, 2023

The pull request was rebased against master so it is easy to review it.

[test]

Copy link
Member

@frenzymadness frenzymadness left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thank you!

@frenzymadness frenzymadness merged commit 50bcaf5 into master Oct 25, 2023
@frenzymadness frenzymadness deleted the support_python311_minimal branch October 25, 2023 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants