Nightly feedstock build failed #230

github-actions · 2024-11-20T11:04:54Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

johnkerl · 2024-11-20T13:27:09Z

#231

jdblischak · 2024-11-20T15:24:20Z

Documenting the errors:

# linux-64
* installing *source* package ‘tiledbsoma’ ...
** using staged installation
** updated src/Makevars for system library via pkg-config
** libs
Error: C++20 standard requested but CXX20 is not defined

# osx-64
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/tiledbsoma_types.h:23:
In file included from /Users/runner/miniforge3/conda-bld/tiledbsoma_1732068244867/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/tiledbsoma:51:
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068244867/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/soma/soma_collection.h:107:32: error: 'path' is unavailable: introduced in macOS 10.15 - see https://conda-forge.org/docs/maintainer/knowledge_base.html#newer-c-features-with-old-sdk
  107 |               std::filesystem::path(uri).filename().string(),  // group name
      |                                ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068244867/_build_env/bin/../include/c++/v1/__filesystem/path.h:383:33: note: 'path' has been explicitly marked unavailable here
  383 | class _LIBCPP_EXPORTED_FROM_ABI path {
      |                                 ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068244867/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/etc/Makeconf:200: RcppExports.o] Error 1
ERROR: compilation failed for package ‘tiledbsoma’

# osx-arm64
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_build_env/bin/../include/c++/v1/__format/format_functions.h:427:15: note: in instantiation of function template specialization 'std::__vformat_to<std::back_insert_iterator<std::string>, char, std::back_insert_iterator<std::__format::__output_buffer<char>>>' requested here
  427 |   return std::__vformat_to(std::move(__out_it), __fmt, __args);
      |               ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_build_env/bin/../include/c++/v1/__format/format_functions.h:458:8: note: in instantiation of function template specialization 'std::vformat_to<std::back_insert_iterator<std::string>>' requested here
  458 |   std::vformat_to(std::back_inserter(__res), __fmt, __args);
      |        ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/library/RcppSpdlog/include/spdl.h:39:86: note: in instantiation of function template specialization 'std::vformat<void>' requested here
   39 |     inline void trace(const char* fmt, Args&&... args ) { RcppSpdlog::log_trace(std::vformat(std::string_view(fmt), std::make_format_args(args...)).c_str()); }
      |                                                                                      ^
./rutilities.h:43:19: note: in instantiation of function template specialization 'spdl::trace<std::string, std::string>' requested here
   43 |             spdl::trace("[config_vector_to_map] adding '{}' = '{}'", std::string(namesvec[i]), std::string(confvec[i]));
      |                   ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_build_env/bin/../include/c++/v1/__charconv/to_chars_floating_point.h:42:1: note: 'to_chars' has been explicitly marked unavailable here
   42 | to_chars(char* __first, char* __last, long double __value, chars_format __fmt);
      | ^
9 errors generated.
make: *** [/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_build_env/lib/R/etc/Makeconf:201: RcppExports.o] Error 1
ERROR: compilation failed for package ‘tiledbsoma’

jdblischak · 2024-11-20T21:05:20Z

Unfortunately I ran out of time today. I applied the update ab4c6f5 from #231 to the nightly build on my fork, but unfortunately it failed with the same errors.

github-actions · 2024-11-21T11:16:10Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-11-21T18:38:34Z

As expected, the R build continues to fail due to C++20 compilation issues.

Today I did another experiment. I purposefully skipped the R client build in order to query the status of the Python build with C++20. Note that this build used the commit single-cell-data/TileDB-SOMA@ea3cd1a that was merged earlier this morning.

First the good news:

C++20 is being used to compile the Python client. Current builds on main (example) use the flag -std=c++17, whereas the nightly build uses -std=c++20
The Python client could be built with C++20 for all architectures (linux-64, osx-64, osx-arm64) and Python versions (3.9, 3.10, 3.11, 3.12)

But then the bad news:

The simple import test fails on both linux-64 and osx-64 (tests aren't run when cross-compiling osx-arm64 on osx-64 architecture).

The backtrace is super long (600+ lines) and only ends because Python core dumps. Below is the beginning and end of the backtrace. Note that I've also replaced the long conda installation path with $PREFIX for readability.

# linux-64 py39

import: 'tiledbsoma'
*** Error in `$PREFIX/bin/python': double free or corruption (out): 0x000055b5a86d6b90 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f8ced766329]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../libtiledbsoma.so(+0x647dd)[0x7f8cecc647dd]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../libtiledbsoma.so(+0x64b0e)[0x7f8cecc64b0e]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../libtiledbsoma.so(+0x1b9275)[0x7f8cecdb9275]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../././libspdlog.so.1.14(_ZN6spdlog6logger13set_formatterESt10unique_ptrINS_9formatterESt14default_deleteIS2_EE+0xc5)[0x7f8ceb2ecd95]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../././libspdlog.so.1.14(_ZN6spdlog6logger11set_patternENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_17pattern_time_typeE+0x4d)[0x7f8ceb2f765d]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../.././libtiledb.so.2.26(+0xc3672a)[0x7f8cec63672a]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../.././libtiledb.so.2.26(+0xc36ae9)[0x7f8cec636ae9]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../.././libtiledb.so.2.26(+0xb27547)[0x7f8cec527547]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../.././libtiledb.so.2.26(+0x1c5240)[0x7f8cebbc5240]
/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7f8cee3e79c3]
/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7f8cee3ec59e]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f8cee3e77d4]
/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7f8cee3ebb8b]
/lib64/libdl.so.2(+0xfab)[0x7f8cedfb8fab]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f8cee3e77d4]
/lib64/libdl.so.2(+0x15ad)[0x7f8cedfb95ad]
/lib64/libdl.so.2(dlopen+0x31)[0x7f8cedfb9041]
$PREFIX/bin/python(+0x1f220c)[0x55b5a719120c]

# omitted hundreds of lines of similar output

7f8cee5fa000-7f8cee5fb000 rw-p 00022000 08:01 5236909                    /usr/lib64/ld-2.17.so
7f8cee5fb000-7f8cee5fc000 rw-p 00000000 00:00 0 
7ffda295a000-7ffda297c000 rw-p 00000000 00:00 0                          [stack]
7ffda29b4000-7ffda29b8000 r--p 00000000 00:00 0                          [vvar]
7ffda29b8000-7ffda29ba000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
/home/conda/feedstock_root/build_artifacts/tiledbsoma_1732208694265/test_tmp/conda_test_runner.sh: line 3:  4617 Aborted                 (core dumped) "$PREFIX/bin/python" -s "/home/conda/feedstock_root/build_artifacts/tiledbsoma_1732208694265/test_tmp/run_test.py"
WARNING: Tests failed for tiledbsoma-py-1.15.0rc3.dev20241119-py39h91c51d3_1.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken

jdblischak · 2024-11-21T19:27:13Z

I searched for the error message from the failed R package build on linux-64: Error: C++20 standard requested but CXX20 is not defined

From what I could find, I think we may need to define CXX (and possibly a few other env vars, similar to what we are already doing for the osx-64 build) in ~/.R/Makevars (Shians/NanoMethViz#32 (comment), microsoft/microsoft-r-open#26 (comment))

johnkerl · 2024-11-21T19:35:00Z

@jdblischak I agree this is probably Makevars / Makevars.in ... I'll look soon

jdblischak · 2024-11-21T19:42:07Z

And for tinkering with the feedstock recipe and/or scripts, you can start from the branch nightly-build. It gets overwritten every night, so if you want your experiments to persist beyond a day, I recommend creating a new branch.

johnkerl · 2024-11-22T01:57:55Z

@jdblischak I believe this will be fixed by #231 which addresses precisely this problem.

I'm looking at #231 (comment) and trying to decide what to sequence to make this all work ... 🤔

johnkerl · 2024-11-22T01:58:40Z

Re #231 (comment) I may be confused but I think we just need to merge #231 to address that bit.

johnkerl · 2024-11-22T02:10:20Z

@jdblischak sorry, I need to sync with you tomorrow ...

github-actions · 2024-11-22T11:15:54Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-11-22T15:38:53Z

I believe this will be fixed by #231 which addresses precisely this problem.

I already tried applying your commit to update build-r-tiledbsoma.sh. That was what I reported above in #230 (comment). Unfortunately it was sufficient to fix the nightly builds.

johnkerl · 2024-11-22T15:42:26Z

OK I'm entirely confused. I'll need to sit down and pencil-and-paper out the repro steps I need to follow to make this happen interactively, debug, and fix. Sorry for the delay.

jdblischak · 2024-11-22T19:16:47Z

Today I setting MACOSX_SDK_VERSION to 13.3, since this is also being explored upstream in single-cell-data/TileDB-SOMA#3361

commit
build

The results:

osx-64: Fixed the R build! And the test section passed too. However, the Python import test still failed as before (see details above in Nightly feedstock build failed #230 (comment))
osx-arm64: The R build still failed with the same error messages: error: 'to_chars' is unavailable: introduced in macOS 13.3. I confirmed that version 13.3 of the SDK was downloaded. Next I'll try setting c_stdlib_version too

jdblischak · 2024-11-22T19:31:00Z

OK I'm entirely confused. I'll need to sit down and pencil-and-paper out the repro steps I need to follow to make this happen interactively, debug, and fix. Sorry for the delay.

I created a dedicated branch for troubleshooting the C++20 compilation errors

https://github.com/TileDB-Inc/tiledbsoma-feedstock/tree/troubleshoot-cpp20

It pulls from the main branch of TileDB-SOMA (to mimic the nightly builds). I applied your fix from #231 and also updated the macOS SDK to 13.3

main...troubleshoot-cpp20

To troubleshoot, you can either push commits directly to this branch (or you can spin off your own branch if you'd prefer). Once we have a working feedstock build, then I'll worry about fixing the setup scripts for the nightly build.

johnkerl · 2024-11-22T19:37:30Z

To troubleshoot

Once we have a working feedstock build

Thanks @jdblischak ! I'll check my notes on how to 'drive' a branch interactively rather than waiting for the nightlies (which is how I usually interact with this repo)

jdblischak · 2024-11-22T20:33:44Z

Next I'll try setting c_stdlib_version too

Quick update. This didn't fix the osx-arm64 build of r-tiledbsoma. And I can tell why now. Despite having conda install version 13.3 of the macOS SDK, the compiler is still passed -mmacosx-version-min=11.0. Thus I think we may need to use wrapper scripts just like we do for osx-64. But unfortunately we'll need to use different wrapper scripts (or use an if/else) because the default conda-forge pin is different for osx-64 (10.3) and osx-arm64 (11.0)

https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/c8f57f58b160967c107a174452b2803dd4505f1e/recipe/conda_build_config.yaml#L21-L22

jdblischak · 2024-11-22T20:35:48Z

Also, another interesting development. After updating the macOS SDK to 13.3, the error message from the failed import test for Python is more informative:

import: 'tiledbsoma'
Traceback (most recent call last):
  File "/Users/runner/miniforge3/conda-bld/tiledbsoma_1732303363846/test_tmp/run_test.py", line 2, in <module>
    import tiledbsoma
  File "$PREFIX/lib/python3.9/site-packages/tiledbsoma/__init__.py", line 166, in <module>
    from ._experiment import Experiment
  File "$PREFIX/lib/python3.9/site-packages/tiledbsoma/_experiment.py", line 18, in <module>
    from ._query import ExperimentAxisQuery
  File "$PREFIX/lib/python3.9/site-packages/tiledbsoma/_query.py", line 48, in <module>
    from somacore.query.query import (
ImportError: cannot import name 'Numpyable' from 'somacore.query.query' ($PREFIX/lib/python3.9/site-packages/somacore/query/query.py)

github-actions · 2024-11-23T11:15:16Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

johnkerl · 2024-11-23T16:38:28Z

@jdblischak re Numpyable:

kerl@arvad[prod][main][SOMA]$ pwd
/Users/kerl/git/single-cell-data/SOMA

kerl@arvad[prod][main][SOMA]$ git diff -r python-v1.0.23 -r python-v1.0.24|grep Numpyable
-            Callable[[_Numpyable], npt.NDArray[np.intp]],
-_Numpyable = Union[pa.Array, pa.ChunkedArray, npt.NDArray[np.int64]]
+Numpyable = Union[pa.Array, pa.ChunkedArray, npt.NDArray[np.int64]]
-    def by_obs(self, coords: _Numpyable) -> npt.NDArray[np.intp]:
+    def by_obs(self, coords: Numpyable) -> npt.NDArray[np.intp]:
-    def by_var(self, coords: _Numpyable) -> npt.NDArray[np.intp]:
+    def by_var(self, coords: Numpyable) -> npt.NDArray[np.intp]:
-def _to_numpy(it: _Numpyable) -> np.ndarray:

So we do need (at least) an update to somacore==1.0.24 ... which is mod I'll push to the troubleshoot-cpp20 branch ...

johnkerl · 2024-11-23T16:48:10Z

Thanks @jdblischak ! I'll check my notes on how to 'drive' a branch interactively rather than waiting for the nightlies (which is how I usually interact with this repo)

Closing the loop on 'how to drive': @jdblischak established in Slack:

Commit and push to the troubleshoot-cpp20 branch
Visit https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary which auto-runs

That's it! I'd failed to realize how much autorun/trigger logic the inestimable John B has set up for this ✨

johnkerl · 2024-11-23T21:15:28Z

Closing the loop on 'how to drive'

Addendum:

Edit locally
./build-locally.py

github-actions · 2024-11-24T11:14:38Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-11-25T11:15:59Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-11-25T16:30:40Z

So we do need (at least) an update to somacore==1.0.24 ... which is mod I'll push to the troubleshoot-cpp20 branch ...

@johnkerl Great insight! With your commit a11e54c, we now have a passing osx-64 build on troubleshoot-cpp20

johnkerl · 2024-11-25T17:16:00Z

Current status:

Code-mod and ./build-locally.py is awesome ✨
Commits on troubleshoot-cpp20: main...troubleshoot-cpp20
Commit 5 a11e54c gets somacore==1.0.24 which as @jdblischak noted above is a good thing
Commits 6 cda0c07 and 7 1c9c11a get me successful compiles -- however link fails with RcppExports.o: relocation R_X86_64_PC32 against symbol R_NilValue' can not be used when making a shared object; recompile with -fPIC`
Using -fPIC fac4c8d got compiles and link successfully, but runtime failure with double free or corruption (out): 0x0000615ad82d2a00
There was a recent heap-corruption PR ([c++] Fix heap management in fastercsx single-cell-data/TileDB-SOMA#3365) worth trying as commit 9 78f4e01 but, alas, to no avail

At this point:

We do need to solve this problem
I'll need to debug more
For the release-1.15 branch @XanthosXanthopoulos please put up a PR on the TileDB-SOMA repo -- for the release-1.15 branch only -- which downgrades from C++20 back to C++17

johnkerl · 2024-11-25T17:17:43Z

Info:

Build log without -fPIC: https://gist.github.com/johnkerl/c519ea45544fec80739b3cfcf4e0a398
Build log with -fPIC: https://gist.github.com/johnkerl/c495714521d2a32b68a381ae490b30d0

jdblischak · 2024-11-25T17:57:29Z

@johnkerl thanks for the updates. Love to see the progress you are making. Just two things to add at the moment:

Commits 6 cda0c07 and 7 1c9c11a get me successful compiles -- however link fails with RcppExports.o: relocation R_X86_64_PC32 against symbol R_NilValue' can not be used when making a shared object; recompile with -fPIC`

Interestingly, your local builds are behaving differently than the Azure builds. For both of the above commits, the r-tiledbsoma builds on Azure continued to fail with the same error message: Error: C++20 standard requested but CXX20 is not defined

There was a recent heap-corruption PR ([c++] Fix heap management in fastercsx single-cell-data/TileDB-SOMA#3365) worth trying as commit 9 78f4e01 but, alas, to no avail

That PR was merged into main 4 days ago in single-cell-data/TileDB-SOMA@f17d55c, so it was already included in all your previous builds on the feedstock branch troubleshoot-cpp.

Your commit 78f4e01 had no effect on the build because we are using git_rev: main as the source in this troubleshooting branch, so the recipe variables version and sha256 are not used to download TileDB-SOMA.

johnkerl · 2024-11-25T18:15:48Z

That PR was merged into main 4 days ago in single-cell-data/TileDB-SOMA@f17d55c, so it was already included in all your previous builds on the feedstock branch troubleshoot-cpp.

Ah, I see my confusion.

In recipe/meta.yaml I've been maintaining separate blocks:

# Post-tag real thing:
source:
  # url: https://github.com/single-cell-data/TileDB-SOMA/archive/{{ version }}.tar.gz
  # sha256: {{ sha256 }}
  # Troubleshoot C++20 by pulling latest from main (just like the nightly builds)
  git_url: https://github.com/single-cell-data/TileDB-SOMA.git
  git_rev: main
  git_depth: -1

# Pre-tag canary "will Conda be green if we release":
#source:
#  git_url: https://github.com/single-cell-data/TileDB-SOMA.git
#  git_rev: 20485c793829a15e27d01bb320bea290c561f46d
#  git_depth: -1
#  # hoping to be 1.15.0rc3 <-- FILL IN HERE

-- and I use the second block for not-tag experiments like this.

Whereas in our troubleshoot_cpp20 you, effectively, put the contents of my second block in the first block. I failed to see the forest for the trees. I'll be sure to read more carefully! :)

github-actions · 2024-11-26T11:15:54Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-11-26T20:15:45Z

Today I did some experiments in centralized-tiledb-nightlies because it is able to build nightly TileDB-SOMA-R with C++20. Below are some potentially useful observations I made about this successful non-conda build setup.

Main observations:

By default the GitHub Actions runner ubuntu-24.04 doesn't set any compiler-related environment variables (CC, CXX, CXX_STD, CXX17FLAGS, CXX20FLAGS), which is an obvious difference from the conda feedstock build
The build ignored the exported CXX20FLAGS in the bash environment. I had to define CXX20FLAGS in ~/.R/Makevars in order to adds the flags to the g++ call and remove the deprecation warnings (TileDB-Inc/centralized-tiledb-nightlies@4de0e59)
Here are the compiler settings for the successful build (I manually added the line breaks for readability). Note that it uses -fpic

g++ -std=gnu++20 \
  -I"/usr/share/R/include" \
  -DNDEBUG \
  -I. \
  -I../inst/include/ \
  -I/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/include \
  -I/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledbsoma/include \
  -I/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledbsoma/include/tiledbsoma \
  -D SPDLOG_USE_STD_FORMAT \
  -I'/usr/lib/R/site-library/Rcpp/include' 
  -I'/usr/lib/R/site-library/RcppSpdlog/include' \
  -I'/usr/lib/R/site-library/RcppInt64/include' \
  -I'/usr/lib/R/site-library/nanoarrow/include' \
  -fpic \
  -Wno-deprecated-declarations -Wno-deprecated \
  -c RcppExports.cpp -o RcppExports.o

Minor observations:

libtiledbsoma builds spdlog as an external project (as opposed to the conda build, which installs a pre-built spdlog from conda-forge)
TileDB-R is compiled with C++17

johnkerl · 2024-11-26T20:22:39Z

libtiledbsoma builds spdlog as an external project (as opposed to the conda build, which installs a pre-built spdlog from conda-forge)

This seems like a big clue as I believe we "really need to" build as an external project -- @nguyenv can you offer any insight?

johnkerl · 2024-11-26T20:24:18Z

I just pushed a commit on troubleshoot-cpp20
main...troubleshoot-cpp20
to test out release-1.15 -- my apologies @jdblischak , I just now saw your #234

jdblischak · 2024-11-26T20:26:52Z

I just pushed a commit on ... to test out release-1.15 -- my apologies ...

No need to apologize! I was busy with other things today, so it took me a while to get to this. The good news is that your commit will pass 😄

jdblischak · 2024-11-26T20:28:38Z

This seems like a big clue as I believe we "really need to" build as an external project

Is there something about C++20 as compared to C++17 that affects the spdlog requirement? We've been installing spdlog from conda-forge for both libtiledb and libtiledbsoma for a while now:

tiledbsoma-feedstock/recipe/meta.yaml

Lines 52 to 53 in fd059f0

    
           - spdlog 
        
           - fmt

johnkerl · 2024-11-26T20:31:51Z

One thing is we don't require fmt anymore (@XanthosXanthopoulos @nguyenv please let me know if there is any reason we do) -- fmt::format was an add-on package which the C++ standard fiiiiinally made built-in via std::format only in C++20.

XanthosXanthopoulos · 2024-11-26T20:52:40Z

With C++20 we are setting SPDLOG_USE_STD_FORMAT when building spdlog to avoid warnings when building with gcc14. Not sure whether or not we will have any issue installing it directly instead of building it though.

johnkerl · 2024-11-26T20:53:54Z

With C++20 we are setting SPDLOG_USE_STD_FORMAT when building spdlog to avoid warnings when building with gcc14. Not sure whether or not we will have any issue installing it directly instead of building it though.

@XanthosXanthopoulos let's please find a way to answer that question outside of the iodiosyncracies of this repo

nguyenv · 2024-11-26T21:39:37Z

This seems like a big clue as I believe we "really need to" build as an external project -- @nguyenv can you offer any insight?

Yeah it seems everything just happened to be compatible and working fine before, but with the move to C++20, it's now revealing issues with this setup. I think we should build spdlog as an external project rather than relying on the conda-forge version.

One thing is we don't require fmt anymore (let me know if there is any reason we do)

No, with C++20 we should remove the fmt requirement.

github-actions · 2024-11-27T11:16:21Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-11-27T18:25:42Z

I think we should build spdlog as an external project rather than relying on the conda-forge version.

If that ends up being required, we may need to add a CMake flag to force building spdlog as an external project. Even if we remove spdlog from the requirements for libtiledbsoma, it will still be installed in the conda environment where it is built:

mamba create --dry-run -n test tiledb | grep -e fmt -e spdlog
## + fmt                             11.0.2  h434a139_0        conda-forge     Cached
## + spdlog                          1.14.1  hed91bc2_1        conda-forge      196kB

github-actions · 2024-11-28T11:04:47Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-11-29T11:04:44Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-11-30T11:04:26Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-01T11:04:31Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-02T11:05:08Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-12-02T21:07:19Z

Today I troubleshooted the linux-64 R build. I used the strategy I had previously identified (#230 (comment)) of defining the CXX-related env vars in a user ~/.R/Makevars. Here is what I learned:

The system-level Makeconf installed with R from conda-forge (installed at $(R RHOME)/etc/Makeconf) defines CXX11, CXX14, and CXX17 related env vars, but the CXX20 ones are blank
Just like I discovered in TileDB-Inc/centralized-tiledb-nightlies@4de0e59, the R build completely ignores global environment variables. They have to be defined in ~/.R/Makevars to have an effect
I was able to fix the compilation and linking by defining the following env vars in ~/.R/Makevars:

CXX20 = $CXX
CXX20FLAGS = $CXXFLAGS -Wno-deprecated-declarations -Wno-deprecated
CXX20PICFLAGS = -fPIC
CXX20STD = -std=c++20

However, now I am getting core dumps with the error double free or corruption (out). This first happened when building the help files, so I passed --no-help to skip executing the \Sexpr{} statements in the documentation .Rd files. We already do this for the cross-compiled osx-arm64 build. But when skipping the help files, it then fails with the same error at the step testing if installed package can be loaded from temporary location, so it appears that simply loading the built package is not possible.
I searched for similar errors in R packages. The most promising Issue I found was this one, double free or corruption (out) theislab/zellkonverter#112, where the user reported the same error when using Seurat in a conda environment

github-actions · 2024-12-03T11:04:45Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

johnkerl · 2024-12-03T14:41:13Z

Thanks @jdblischak ! The double-free issue jives with what I saw here:
#230 (comment)

I really appreciate your deep-dive into this:

The system-level Makeconf installed with R from conda-forge (installed at $(R RHOME)/etc/Makeconf) defines CXX11, CXX14, and CXX17 related env vars, but the CXX20 ones are blank

-- this explains so much!

jdblischak · 2024-12-03T20:24:59Z

The double-free issue jives with what I saw here

Nice! Thanks for the reminder.

So the good news is that we are both observing consistent behavior:

Compilation is fixed by defining CXX20 and CXX20STD
Linking is fixed by adding the flag -fPIC
Package fails to load with double free or corruption (out)

Interestingly, searching this long thread, I had also previously observed the same error message when importing the Python package on linux-64 (#230 (comment)).

So while all troubleshooting routes are leading us to the same runtime error, which is nice for consistency, the bad news is that we don't know what is causing the error (speaking for myself at least, I'm at a loss of what additional build options to tweak).

johnkerl · 2024-12-03T20:37:16Z

@jdblischak what I'd love to get is an interactive (non-CI) DOcker build that repros the double-free error. (I haven't succeeded.)

jdblischak · 2024-12-03T20:41:26Z

what I'd love to get is an interactive (non-CI) DOcker build that repros the double-free error

Good idea. I think this should be feasible. If I can get it working, I'll send you instructions

github-actions bot added the nightly-failure Nightly Azure build failed label Nov 20, 2024

github-actions bot assigned johnkerl and jdblischak Nov 20, 2024

jdblischak mentioned this issue Nov 20, 2024

Build r-tiledbsoma with C++ 20 #231

Open

jdblischak mentioned this issue Nov 21, 2024

[c++/python] Update minimum macOS version to 13.3 single-cell-data/TileDB-SOMA#3361

Open

jdblischak mentioned this issue Nov 26, 2024

Reminder: update requirements for 1.15.0 #233

Open

Nightly feedstock build failed #230

Nightly feedstock build failed #230

Comments

github-actions bot commented Nov 20, 2024

johnkerl commented Nov 20, 2024

jdblischak commented Nov 20, 2024

jdblischak commented Nov 20, 2024

github-actions bot commented Nov 21, 2024

jdblischak commented Nov 21, 2024

jdblischak commented Nov 21, 2024

johnkerl commented Nov 21, 2024

jdblischak commented Nov 21, 2024

johnkerl commented Nov 22, 2024

johnkerl commented Nov 22, 2024 • edited Loading

johnkerl commented Nov 22, 2024

github-actions bot commented Nov 22, 2024

jdblischak commented Nov 22, 2024

johnkerl commented Nov 22, 2024

jdblischak commented Nov 22, 2024

jdblischak commented Nov 22, 2024

johnkerl commented Nov 22, 2024

jdblischak commented Nov 22, 2024

jdblischak commented Nov 22, 2024

github-actions bot commented Nov 23, 2024

johnkerl commented Nov 23, 2024 • edited Loading

johnkerl commented Nov 23, 2024

johnkerl commented Nov 23, 2024

github-actions bot commented Nov 24, 2024

github-actions bot commented Nov 25, 2024

jdblischak commented Nov 25, 2024

johnkerl commented Nov 25, 2024

johnkerl commented Nov 25, 2024

jdblischak commented Nov 25, 2024

johnkerl commented Nov 25, 2024

github-actions bot commented Nov 26, 2024

jdblischak commented Nov 26, 2024

johnkerl commented Nov 26, 2024

johnkerl commented Nov 26, 2024

jdblischak commented Nov 26, 2024

jdblischak commented Nov 26, 2024

johnkerl commented Nov 26, 2024

XanthosXanthopoulos commented Nov 26, 2024

johnkerl commented Nov 26, 2024

nguyenv commented Nov 26, 2024

github-actions bot commented Nov 27, 2024

jdblischak commented Nov 27, 2024

github-actions bot commented Nov 28, 2024

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 30, 2024

github-actions bot commented Dec 1, 2024

github-actions bot commented Dec 2, 2024

jdblischak commented Dec 2, 2024

github-actions bot commented Dec 3, 2024

johnkerl commented Dec 3, 2024

jdblischak commented Dec 3, 2024

johnkerl commented Dec 3, 2024

jdblischak commented Dec 3, 2024

johnkerl commented Nov 22, 2024 •

edited

Loading

johnkerl commented Nov 23, 2024 •

edited

Loading