Update RAPIDS Python packages to use rapids-build-backend #31

vyasr · 2024-03-13T17:09:15Z

rapids-build-backend is a wrapper around standard backends like scikit-build-core and setuptools that handles some of the standard issues that we face for RAPIDS packages (CUDA versioning, alpha versions, etc). Substituting it into existing RAPIDS Python packages should be fairly painless, but will require some careful testing to verify that nothing is broken.

The backend need not be updated in lockstep across all of RAPIDS for this to work, and merging PRs in any order should be generally safe. However, since there are cases like unified devcontainers where underlying build commands may need an update, it would be best to try and test at least a couple of core packages together to verify that everything works as expected.

Changes to `rapids-build-backend` and `rapids-dependency-file-generator`

Give feedback

Automatically write GIT_COMMIT file (Automatically figure out where to write GIT_COMMIT rapids-build-backend#30)
Add disable-cuda option (Add disable-cuda option rapids-build-backend#32)
Warn if dependencies.yaml is missing (warn if dependencies file does not exist rapids-build-backend#33)
Release version 0.2.0 (Version 0.2.0 rapids-build-backend#34)
Prefix -C options with rapidsai. (Prefix and filter command line options with rapidsai. rapids-build-backend#37)
support projects with non-trivial setup.py ([RFC] handling CUDA-suffixed dependencies needed at setuptools.setup() time rapids-build-backend#39)
set up publishing to public PyPI (Enable release wheels publish to pypi rapids-build-backend#47)
Options

Other

Give feedback

Hook to verify alpha spec in dependencies.yaml (Add verify-alpha-spec hook pre-commit-hooks#28)
Any necessary rapidsai/devcontainer adjustments (support rapids-build-backend devcontainers#325)
replace "commit-file" with "commit-files" (or just remove that configuration) in projects that were updated before Automatically figure out where to write GIT_COMMIT rapids-build-backend#30
remove .gitattributes files left behind from when projects used versioneer (Update RAPIDS Python packages to use rapids-build-backend #31 (comment))
Options

The text was updated successfully, but these errors were encountered:

Reading through this project's code for work on rapidsai/build-planning#31, I noticed it isn't using type hints or running a type-checking This proposes the following: * adding `mypy` to checks run via pre-commit * adding type hints for a few cases I think this'll make the code a little easier to understand, and improve the change of catching some types of bugs (like treating a value which can be `None` as if it was unconditionally not `None) during development. ### How I tested this ```shell pre-commit run --all-files ```

mmccarty · 2024-05-15T16:55:31Z

Can we get a task list?

jakirkham · 2024-05-15T17:00:29Z

This was originally proposed for RAPIDS v24.06.00. However in discussion in our build-infra meeting today, we decided it is probably too late for this to go into v24.06.00. So moving this to v24.08.00

## Description Contributes to rapidsai/build-planning#31. Related to #24. This proposes the following changes: * if the file indicated by `dependencies-file` does not exist, emit a warning ## Notes for Reviewers ### Why add this warning? While testing rapidsai/cudf#15245, I observed `rapids-build-backend==0.1.1` failing to update the build dependencies of wheels. This showed up with errors like the following: ```text ERROR: Could not find a version that satisfies the requirement rmm==24.6.* (from versions: 0.0.1) ``` It took me maybe an hour to figure out what was going on.... I wasn't setting `dependencies-file` in the `[tool.rapids-build-backend]` configuration. `cudf` is laid out like this: ```text cudf/ |___python/ |_____cudf/ |_________pyproject.toml |_____dask_cudf/ |_________pyproject.toml ``` Not setting `dependencies-file` meant that `rapids-build-backend` defaulted to `"dependencies.yaml"` (e.g. `cudf/python/cudf/dependencies.yaml`). That file doesn't exist, and so it just silently skipped all `dependencies.yaml`-related stuff via this: https://github.com/rapidsai/rapids-build-backend/blob/97a19504201f18cfae1864c8be99c1d73cbcc0f9/rapids_build_backend/impls.py#L170-L175 With the warning added in this PR, I would have at least had a hint from CI logs pointing me at the potential problem. ### ~Why change the default?~ <details><summary>edited: removed (click me)</summary> `"./dependencies.yaml"` (e.g., `dependencies.yaml` located next to the wheel's `pyproject.toml`) is not a working default for any RAPIDS project, as far as I know. In my experience, most of them are either laid out like `cuml`: ```text cuml/ |____dependencies.yaml |____python/ |_____pyproject.toml ``` Or like `rmm` ```text rmm/ |____dependencies.yaml |____python/ |_____librmm |_____pyproject.toml |_____rmm |______pyproject.toml ``` See https://github.com/search?q=%22%5Bproject%5D%22+org%3Arapidsai+path%3Apyproject.toml&type=code. Changing to `../dependencies.yaml` would at least automatically work for some projects (like `cuml`) with minimal risk of reaching up outside of the repo to some other project's `dependencies.yaml` (which would be more of a risk with, say, `../../dependencies.yaml`. </details> ### Could we just make this a hard error instead of a warning? I think we should! I'd personally support being even stricter at this point in development and making an existing `dependencies.yaml` file a hard requirement for running `rapids-build-backend`, meaning: * no default value for `dependencies-file` * loud runtime error if the file pointed to by `dependencies-file` doesn't exist If we can't think of an existing case that requires running this without a `dependencies.yaml`, then I think we should implement this stricter interface. Making something more flexible later is always easier than making it stricter, and this strictness right now will save some developer time (e.g. would have saved me some time working on `cudf` today).

Contributes to rapidsai/build-planning#31 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Corey J. Nolet (https://github.com/cjnolet) URL: #2331

This PR uses `rapids-build-backend` to simplify wheel builds and reduce the complexity of various CI/build scripts. See also: - https://github.com/rapidsai/rapids-build-backend - rapidsai/build-planning#31 - rapidsai/cudf#15245 - #1529 Authors: - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Vyas Ramasubramani (https://github.com/vyasr) URL: #1502

jameslamb · 2024-05-28T16:48:55Z

Can we get a task list?

I've updated the list of migrations with what I think is a complete set of projects we want to roll this out to. So we can track what and how much work remains to complete this.

Included any project in the rapidsai organization for which at least one of the following is true:

publishes wheels with CUDA suffix
has dependencies with CUDA suffix
set __git_commit__ in Python module-level constant
uses a build script to add alpha specs to RAPIDS dependencies

Some that don't have a dependencies.yaml but where I still think we want rapids-build-backend to handle CUDA suffixes for the project name and populating __git_commit__:

This PR uses `rapids-build-backend` to simplify wheel builds and reduce the complexity of various CI/build scripts. See also: - https://github.com/rapidsai/rapids-build-backend - rapidsai/build-planning#31 Authors: - Vyas Ramasubramani (https://github.com/vyasr) - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) Approvers: - Lawrence Mitchell (https://github.com/wence-) - Bradley Dice (https://github.com/bdice) URL: #15245

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1364

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #1627

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1057

Contributes to rapidsai/build-planning#31 Follow-up to #15245 RAPIDS DLFW builds prefer to build all RAPIDS packages together without CUDA suffixes, leading to the following set of requirements for `cudf` wheels built there: * project name must be `cudf` (not `cudf-cu12`) * all dependencies must be unsuffixed (e.g. `rmm` not `rmm-cu12`) * the correct set of dependencies based on CUDA version must be expressed in the wheel metadata (e.g. `cubinlinker` and `ptxcompiler` on CUDA 11, `pynvjitlink` on CUDA 12) To meet all 3 of those, this proposes decomposing CUDA-suffixed dependencies in `dependencies.yaml` into two lists... `cuda_suffixed="true"` and `cuda_suffixed="false"`. That'd allow DLFW builds to do the following to meet its requirements: ```shell pip wheel \ -C rapidsai.disable-cuda=true \ -C rapidsai.matrix-entry="cuda=12.5;cuda_suffixed=false" \ . ``` Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: #16183

…te-version.sh (#1367) Contributes to rapidsai/build-planning#31. Follow-up to #1364. Implements some of the suggestions made in rapidsai/cudf#16183 (after #1364 was already merged): * removing `cuda_suffixed: "false"` blocks in `dependencies.yaml` wherever they're identical to each other and the fallback matrix * changing `dependencies.yaml` anchors with names like `*_conda` to `*_unsuffixed`, to reflect the fact that they're not conda-specific * checking that `update-version.sh` catches all changes to versions ## Notes for Reviewers ### How I tested this Looked for `update-versions.sh` issues manually like this: ```shell git fetch upstream --tags ci/release/update-version.sh '24.10.0' git grep -E '24\.8|24\.08|0\.39' ``` The did find a few problems (like UCX dependency versions not being updated). This fixes those issues. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: #1367

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Bradley Dice (https://github.com/bdice) URL: #2388

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #411

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #609

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #251

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #1414

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #5974

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) - Paul Taylor (https://github.com/trxcllnt) Approvers: - Bradley Dice (https://github.com/bdice) URL: #247

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 ## Notes for Reviewers ### Why target 24.08? This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #4552

Contributes to rapidsai/build-planning#31 In short, RAPIDS DLFW builds want to produce wheels with unsuffixed dependencies, e.g. `cudf` depending on `rmm`, not `rmm-cu12`. This PR is part of a series across all of RAPIDS to try to support that type of build by setting up CUDA-suffixed and CUDA-unsuffixed dependency lists in `dependencies.yaml`. For more details, see: * rapidsai/build-planning#31 (comment) * rapidsai/cudf#16183 This is targeting 24.08 because: 1. it should be very low-risk 2. getting these changes into 24.08 prevents the need to carry around patches for every library in DLFW builds using RAPIDS 24.08 Authors: - James Lamb (https://github.com/jameslamb) - Paul Taylor (https://github.com/trxcllnt) Approvers: - Bradley Dice (https://github.com/bdice) URL: rapidsai#247

commit 606d15e7260b553cbdb69f9ecd935c12ba94e430 Author: Thomas Li <[email protected]> Date: Wed Jul 31 14:30:48 2024 +0000 put back mistakenly removed CMakeLists.txt commit feac68de39be09c1751d0ccc2bb5f93b1075ac8f Author: Thomas Li <[email protected]> Date: Wed Jul 31 13:59:50 2024 +0000 rpath was the problem? commit b2b68e14b9faa1dac0f2516667f65ecb5693a744 Author: Thomas Li <[email protected]> Date: Tue Jul 30 22:29:14 2024 +0000 maybe fix? commit 5243eac8a90114e4fdf794760cb6b6029d9ba1a1 Author: Thomas Li <[email protected]> Date: Tue Jul 30 21:11:03 2024 +0000 fix cuda suffixing commit acb31227d3ffb07e4a35be5d1c0ec6cbadbfe53d Author: Thomas Li <[email protected]> Date: Tue Jul 30 20:29:52 2024 +0000 fixes commit b2306df549ac5db08dc0d1b09df270137dacfe9d Author: Thomas Li <[email protected]> Date: Tue Jul 30 20:08:13 2024 +0000 fixes commit d6d91df1510a70d79fefacf8b57ca1caf027edf8 Merge: b7a2782f1a 7b3e73a7e3 Author: Thomas Li <[email protected]> Date: Tue Jul 30 19:32:18 2024 +0000 Merge branch 'branch-24.10' of github.com:rapidsai/cudf into setup-pylibcudf-package commit 7b3e73a7e38b671db1387879cfa963fe61060c36 Merge: ce259fff66 dbf4bd02a8 Author: gpuCI <[email protected]> Date: Tue Jul 30 13:14:19 2024 -0400 Merge pull request #16435 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit dbf4bd02a8fdccd1891edbc2d049c3ddddb234b3 Author: GALI PREM SAGAR <[email protected]> Date: Tue Jul 30 12:14:14 2024 -0500 Add about rmm modes in `cudf.pandas` docs (#16404) This PR adds user facing docs for rmm memory modes and prefetching. --------- Co-authored-by: Mark Harris <[email protected]> Co-authored-by: Bradley Dice <[email protected]> commit ce259fff6641dd847883d535645c7c17c36fb7ec Merge: b8bfe2c912 0f07b0bb5e Author: gpuCI <[email protected]> Date: Tue Jul 30 09:02:26 2024 -0400 Merge pull request #16433 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 0f07b0bb5e2cc89ca66e9d9639ff6ac961ec0471 Author: GALI PREM SAGAR <[email protected]> Date: Tue Jul 30 08:02:21 2024 -0500 Enable prefetching before `runpy` (#16427) This PR enables prefetching before we execute the `runpy` module and script code. commit b8bfe2c91234032cbe9b2549e46a08109e238c8a Merge: d1be0b6dc0 5feeaf3827 Author: gpuCI <[email protected]> Date: Tue Jul 30 09:02:06 2024 -0400 Merge pull request #16432 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 5feeaf3827bfd20755cdd0516ef0c6ba484a600c Author: Richard (Rick) Zamora <[email protected]> Date: Tue Jul 30 08:02:01 2024 -0500 [Bug] Remove loud `NativeFile` deprecation noise for `read_parquet` from S3 (#16415) Important follow-up to https://github.com/rapidsai/cudf/pull/16132 Without this PR, using `dask_cudf.read_parquet("s3://...", ...)` will result in loud deprecation warnings after `compute`/`persist` is called. This is because dask will always pass `NativeFile` objects down to cudf. My fault for missing this earlier! commit d1be0b6dc06fddd0b69fb69731281b16894cb132 Author: Matthew Roeschke <[email protected]> Date: Mon Jul 29 15:12:38 2024 -1000 Align CategoricalIndex APIs with pandas 2.x (#16369) Mostly exposing methods that were available on the CategoricalColumn Authors: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16369 commit 368a34ca9fd7db1b6cfb6e7817978e3e4fcfb00b Author: Bradley Dice <[email protected]> Date: Mon Jul 29 20:05:17 2024 -0500 Use RMM adaptor constructors instead of factories. (#16414) This PR uses RMM memory resource adaptor constructors instead of factory functions. With CTAD, we do not need the factory and can use the constructor directly. The factory will be deprecated in https://github.com/rapidsai/rmm/pull/1626. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Nghia Truong (https://github.com/ttnghia) - Jayjeet Chakraborty (https://github.com/JayjeetAtGithub) URL: https://github.com/rapidsai/cudf/pull/16414 commit e8048f7f3d66433203651a6a603d4de1360ca5ca Merge: f8eb63e499 bd302d773c Author: gpuCI <[email protected]> Date: Mon Jul 29 20:07:38 2024 -0400 Merge pull request #16431 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit bd302d773c50552531bc7f11f782f8ed876e8fab Author: Nghia Truong <[email protected]> Date: Mon Jul 29 17:07:33 2024 -0700 Support thread-safe for `prefetch_config::get` and `prefetch_config::set` (#16425) This adds muti-thread support for `prefetch_config` getter and setter functions. This avoid the issue that the config map is corrupted in multi-thread environments. Closes https://github.com/rapidsai/cudf/issues/16426. --------- Signed-off-by: Nghia Truong <[email protected]> commit f8eb63e499f94d583d715f5c1f5e6f234589be57 Author: Matthew Roeschke <[email protected]> Date: Mon Jul 29 12:39:19 2024 -1000 Align Index APIs with pandas 2.x (#16361) Similar to https://github.com/rapidsai/cudf/pull/16310, the follow APIs have been modified to adjust/add parameters * `to_flat_index` * `isin` * `unique` * `transpose` Authors: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16361 commit 743e16426c564d0ed0d7e3d9be5f67e4605c4f32 Author: James Lamb <[email protected]> Date: Mon Jul 29 14:19:43 2024 -0500 update some branch references in GitHub Actions configs (#16397) Fixes some lingering references to `branch-24.08` in the `pr_issue_status_automation` CI workflow. This was missed when new branches were cut because that file ends in `.yml` and `update-version.sh` was only modifying files ending in `.yaml`. The corresponding `update-version.sh` changes were made in #16183 and are already on 24.10 thanks to forward mergers. https://github.com/rapidsai/cudf/blob/dc05a01f3fc0742c5fbbddd86a0f2007bfdc2050/ci/release/update-version.sh#L78 ## Notes for Reviewers I checked like this, and don't see any other missed references: ```shell git grep -E '24\.8|24\.08|0\.39' ``` Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: https://github.com/rapidsai/cudf/pull/16397 commit 35796057b64e258713d4d89ba368837d30a1a9c5 Author: Matthew Roeschke <[email protected]> Date: Mon Jul 29 08:33:23 2024 -1000 Align misc DataFrame and MultiIndex methods with pandas 2.x (#16402) The API changes in this PR are mostly adding implementations or adding missing keyword argument (although they might not be implemented). The APIs affected are: * `DataFrame.insert` * `DataFrame.melt` * `DataFrame.merge` * `DataFrame.quantile` * `DataFrame.cov` * `DataFrame.corr` * `DataFrame.median` * `DataFrame.rolling` * `DataFrame.resample` * `DataFrame.dropna` * `MultiIndex.from_tuple` * `MultiIndex.from_frame` * `MultiIndex.from_product` Authors: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16402 commit 6e7624d6b31c93b0547590929ac63ed8e3a48d24 Author: David Wendt <[email protected]> Date: Mon Jul 29 14:06:51 2024 -0400 Add stream parameter to reshape APIs (#16410) Adds `stream` parameter to reshape APIs: - `cudf::interleave_columns` - `cudf::tile` - `cudf::byte_cast` Found while working #15983 Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Bradley Dice (https://github.com/bdice) - Nghia Truong (https://github.com/ttnghia) URL: https://github.com/rapidsai/cudf/pull/16410 commit 58f47242fe04b1e25fd42e1e45e8c15417140777 Author: Matthew Roeschke <[email protected]> Date: Mon Jul 29 06:09:21 2024 -1000 Align groupby APIs with pandas 2.x (#16403) The following breaking APIs are affected: * `apply` * `transform` * `describe` The rest of the APIs are non-breaking and generally will raise a `NotImplementedError` Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16403 commit 18c1465b597284d8b558964cc0ca48de7da60a17 Author: Matthew Roeschke <[email protected]> Date: Mon Jul 29 06:06:07 2024 -1000 Align ewm APIs with pandas 2.x (#16413) These all currently are not implemented and raise a `NotImplementedError` Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16413 commit eed0b1f36c84aa4a4bf17a3b99f931940cb6ddd9 Merge: 24997fda19 a51964ed8b Author: gpuCI <[email protected]> Date: Mon Jul 29 09:42:33 2024 -0400 Merge pull request #16419 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit a51964ed8b00c3c88d463e329af7ec8378642343 Author: GALI PREM SAGAR <[email protected]> Date: Mon Jul 29 08:42:27 2024 -0500 Fix a `pandas-2.0` missing attribute error (#16416) `NumpyEADtype` is a 2.1.0+ change, this PR handles the missing attribute error in pandas-2.0 commit 24997fda194d5b8af34048a8bf275830cabbff8c Author: Muhammad Haseeb <[email protected]> Date: Fri Jul 26 18:37:30 2024 -0700 Deduplicate decimal32/decimal64 to decimal128 conversion function (#16236) Closes #16194 This PR deduplicates the `convert_data_to_decimal128` function from `to_arrow.cu`, `writer_impl.cu` and `to_arrow_device.cu` to a common location. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Nghia Truong (https://github.com/ttnghia) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16236 commit 473dec55abd1a3d9d540c541443f831d18ebb532 Author: Jayjeet Chakraborty <[email protected]> Date: Fri Jul 26 14:45:12 2024 -0700 Add query 10 to the TPC-H suite (#16392) Adds Q10 to the TPC-H benchmark suite Authors: - Jayjeet Chakraborty (https://github.com/JayjeetAtGithub) Approvers: - Mike Wilson (https://github.com/hyperbolic2346) - Yunsong Wang (https://github.com/PointKernel) URL: https://github.com/rapidsai/cudf/pull/16392 commit 46ff702144a2477d06ffabd3d92d38967c10b1ff Merge: 73158f06e2 5dd3efba5b Author: gpuCI <[email protected]> Date: Fri Jul 26 16:47:54 2024 -0400 Merge pull request #16411 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 5dd3efba5b7e0c22dce87cf20aecb1b198677d2e Author: David Wendt <[email protected]> Date: Fri Jul 26 16:47:49 2024 -0400 Fix nightly memcheck error for empty STREAM_INTEROP_TEST (#16406) ## Description The `STREAM_INTEROP_TEST` code was commented out in #16379 so the `compute-sanitizer` returns an error for this test in the nightly cpp-memcheck tests. https://github.com/rapidsai/cudf/actions/runs/10107041505/job/27950193878#step:9:62177 This PR comments out the empty test so it is not built. The test will be re-enabled in a future release when the deprecated functions are replaced. ## Checklist - [x] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [x] New or existing tests cover these changes. - [x] The documentation is up to date with these changes. commit 73158f06e2b816d88e4a2b71f236812ab997391f Merge: dc05a01f3f f88a242832 Author: Jake Awe <[email protected]> Date: Fri Jul 26 13:14:22 2024 -0500 Merge pull request #16409 from vyasr/branch-24.10-merge-branch-24.08 Branch 24.10 merge branch 24.08 commit f88a242832a1c991c615961631f02c9875ab871f Merge: dc05a01f3f cd762b4eb1 Author: Vyas Ramasubramani <[email protected]> Date: Fri Jul 26 18:10:32 2024 +0000 Merge branch 'branch-24.08' into branch-24.10-merge-branch-24.08 commit cd762b4eb1fd55a0bc5079ed69bfc04426f10e60 Author: Matthew Roeschke <[email protected]> Date: Fri Jul 26 08:08:01 2024 -1000 Gate ArrowStringArrayNumpySemantics cudf.pandas proxy behind version check (#16401) ## Description `ArrowStringArrayNumpySemantics` was newly added in 2.1: https://github.com/pandas-dev/pandas/blob/2.1.x/pandas/core/arrays/string_arrow.py#L488, so putting the proxy wrapper behind a version check for pandas 2.0.x compat ```ipython In [1]: %load_ext cudf.pandas In [2]: import pandas as pd In [3]: pd.__version__ Out[3]: '2.0.0' ``` ## Checklist - [ ] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [ ] New or existing tests cover these changes. - [ ] The documentation is up to date with these changes. commit 1cea1eaf6c1e87e65729897dd9bbedc4bdc5e7ab Author: Kyle Edwards <[email protected]> Date: Thu Jul 25 16:26:34 2024 -0400 Don't export bs_thread_pool (#16398) ## Description cudf does not currently export any headers that depend on bs_thread_pool, and having it as a dependency is currently causing problems for consumers. Avoid exporting it since it's not needed. ## Checklist - [ ] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [ ] New or existing tests cover these changes. - [ ] The documentation is up to date with these changes. commit dc05a01f3fc0742c5fbbddd86a0f2007bfdc2050 Merge: fb2021fe82 e553295cfa Author: gpuCI <[email protected]> Date: Thu Jul 25 12:14:52 2024 -0400 Merge pull request #16396 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit e553295cfaf2f5bd1f539ee78d9a3a064e00e5f0 Author: brandon-b-miller <[email protected]> Date: Thu Jul 25 11:14:47 2024 -0500 Require fixed width types for casting in `cudf-polars` (#16381) Fixes a bug where numeric <-> string casts are not being properly rejected at the cudf-polars level. Authors: - https://github.com/brandon-b-miller Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16381 commit fb2021fe82724746ae1c58345ed37f7e7a0207ed Merge: 673b96f6d1 f756e01a3c Author: Ray Douglass <[email protected]> Date: Thu Jul 25 11:06:30 2024 -0400 Merge pull request #16391 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit f756e01a3c5ff83421b1afb44460d9e5147a410e Author: Thomas Li <[email protected]> Date: Thu Jul 25 07:04:47 2024 -0700 Implement support for scan_ndjson in cudf-polars (#16263) Implement support for scan_ndjson in cudf-polars. Authors: - Thomas Li (https://github.com/lithomas1) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/16263 commit 673b96f6d15dbd5d8bcb22d612d3c324aa899e26 Merge: 5a3399bec8 4cc37896a5 Author: Jake Awe <[email protected]> Date: Thu Jul 25 08:27:15 2024 -0500 Merge pull request #16393 from jameslamb/branch-24.10-merge-branch-24.08 Merge branch-24.08 into branch-24.10 commit d953676e9281125a5b8bd9be739c997611471771 Author: Robert Maynard <[email protected]> Date: Thu Jul 25 04:49:12 2024 -0400 Hide visibility of non public symbols (#15982) Converts cudf over to a system of explicit markup of what symbols should be used by consumers. This is done by compiling with `-fvisibility=hidden` and explicit markup via `CUDF_EXPORT` of components we want usable. Due to issues with tests a portion of `include/` detail functions had to be marked as public API. More concernning are that the tests leverage functions from `cpp/` that are never part of the installed headers. That set of files can be found at https://github.com/rapidsai/cudf/commit/16b365635ab0f86bb1cc6db5f036564e8290f3b1 and we should discuss how we should restructure cudf to remove these. Authors: - Robert Maynard (https://github.com/robertmaynard) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) - Nghia Truong (https://github.com/ttnghia) URL: https://github.com/rapidsai/cudf/pull/15982 commit 4aefcc7b2988346166b9a757fc837e93f6f0a3bb Author: GALI PREM SAGAR <[email protected]> Date: Wed Jul 24 22:30:35 2024 -0500 Add ability to prefetch in `cudf.pandas` and change default to managed pool (#16296) This PR adds ability to prefetch in `cudf.pandas` based off of: https://github.com/rapidsai/rmm/pull/1608/ Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) - Muhammad Haseeb (https://github.com/mhaseeb123) - Vyas Ramasubramani (https://github.com/vyasr) - Mark Harris (https://github.com/harrism) URL: https://github.com/rapidsai/cudf/pull/16296 commit 6486bb928dfb0e1817b0604572e2f5789d05c596 Author: Matthew Murray <[email protected]> Date: Wed Jul 24 22:24:46 2024 -0400 Migrate lists/filtering to pylibcudf (#16184) Apart of #15162 Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16184 commit a33f520b370d048a22de031294311c241ab23858 Author: David Gardner <[email protected]> Date: Wed Jul 24 18:42:16 2024 -0700 Fix inconsistent usage of 'results' and 'records' in read-json.md (#15766) * Fix inconsistent usage of 'results' and 'records' in `docs/cudf/source/user_guide/io/read-json.md` Authors: - David Gardner (https://github.com/dagardner-nv) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - Nghia Truong (https://github.com/ttnghia) URL: https://github.com/rapidsai/cudf/pull/15766 commit 5a3399bec868f44d13c003f172c665919096d8e8 Author: James Lamb <[email protected]> Date: Wed Jul 24 19:26:12 2024 -0500 fix [tool.setuptools] reference in custreamz config (#16365) Noticed this warning in logs from #16183 > _/python3.10/site-packages/setuptools/config/pyprojecttoml.py:70: _ToolsTypoInMetadata: Ignoring [tools.setuptools] in pyproject.toml, did you mean [tool.setuptools]?_ This fixes that. ## Notes for Reviewers Intentionally targeting this at 24.10. This misconfiguration has been in `custreamz` since the 23.04 release ([git blame link](https://github.com/rapidsai/cudf/blame/e6d412cba7c23df7ee500c28257ed9281cea49b9/python/custreamz/pyproject.toml#L60)). I think the only effect might be that some test files are included in wheels when we don't want to. I don't think the fix for it needs to be rushed into 24.08. I searched across RAPIDS in case this was copied from somewhere else... don't see any other instances of this typo that need to be fixed. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16365 commit 4cc37896a5dff1e019f0dff8101f3a84a05fd5d8 Merge: 29ce5c529e a36dacb663 Author: James Lamb <[email protected]> Date: Wed Jul 24 18:54:56 2024 -0500 Merge branch-24.08 into branch-24.10 commit a36dacb66325e03d3264482d35a5cf7e0b6c7a37 Author: Lawrence Mitchell <[email protected]> Date: Thu Jul 25 00:31:40 2024 +0100 Make C++ compilation warning free after #16297 (#16379) In https://github.com/rapidsai/cudf/pull/16297, we deprecated the use of `to_arrow` in favour of `to_arrow_host` and `to_arrow_device`. However, the scalar detail overload of `to_arrow` used the public table overload. So we get a warning when compiling internal libcudf code. Fix this by using the detail API, and fix a bug along the way where we were not passing through the arrow memory resource. Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - David Wendt (https://github.com/davidwendt) - Michael Schellenberger Costa (https://github.com/miscco) - Vyas Ramasubramani (https://github.com/vyasr) - Karthikeyan (https://github.com/karthikeyann) URL: https://github.com/rapidsai/cudf/pull/16379 commit ae4c7e3ce4fe100eb919ca00fa34461e44078ba9 Author: James Lamb <[email protected]> Date: Wed Jul 24 18:30:53 2024 -0500 split up CUDA-suffixed dependencies in dependencies.yaml (#16183) Contributes to https://github.com/rapidsai/build-planning/issues/31 Follow-up to #15245 RAPIDS DLFW builds prefer to build all RAPIDS packages together without CUDA suffixes, leading to the following set of requirements for `cudf` wheels built there: * project name must be `cudf` (not `cudf-cu12`) * all dependencies must be unsuffixed (e.g. `rmm` not `rmm-cu12`) * the correct set of dependencies based on CUDA version must be expressed in the wheel metadata (e.g. `cubinlinker` and `ptxcompiler` on CUDA 11, `pynvjitlink` on CUDA 12) To meet all 3 of those, this proposes decomposing CUDA-suffixed dependencies in `dependencies.yaml` into two lists... `cuda_suffixed="true"` and `cuda_suffixed="false"`. That'd allow DLFW builds to do the following to meet its requirements: ```shell pip wheel \ -C rapidsai.disable-cuda=true \ -C rapidsai.matrix-entry="cuda=12.5;cuda_suffixed=false" \ . ``` Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16183 commit 29ce5c529ea9ea18edc32ab905f1ef076f266008 Author: Michael Schellenberger Costa <[email protected]> Date: Thu Jul 25 01:29:41 2024 +0200 Fix some issues with deprecated / removed cccl facilities (#16377) `cub::If` has been deprecated and should not be used. There is a better alternative in `cuda::std::conditional_t` `thrust::{binary, unary}_function` has been deprecated and does not serve a purpose similar to the removed `std::{binary, unary}_function` Rather than relying on the type aliases one should use the `std::invoke` machinery Authors: - Michael Schellenberger Costa (https://github.com/miscco) Approvers: - Bradley Dice (https://github.com/bdice) - Nghia Truong (https://github.com/ttnghia) - Bernhard Manfred Gruber (https://github.com/bernhardmgruber) URL: https://github.com/rapidsai/cudf/pull/16377 commit a6b1cf1fa96d622626a9e4d99a5c71d33fb1bd49 Merge: 2eabe0de58 59f65843b8 Author: gpuCI <[email protected]> Date: Wed Jul 24 19:10:33 2024 -0400 Merge pull request #16389 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 59f65843b80d967f743841aee8489b6ae63b269a Author: Muhammad Haseeb <[email protected]> Date: Wed Jul 24 16:10:28 2024 -0700 Gracefully CUDF_FAIL when `skip_rows > 0` in Chunked Parquet reader (#16385) This PR must merge in cudf 24.08 to avoid unhandled expections. Gracefully CUDF_FAIL in chunked parquet reader when `skip_rows>0` which may result in runtime exceptions like segfaults or an infinite loop. See #16186 for more information. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - David Wendt (https://github.com/davidwendt) - Vyas Ramasubramani (https://github.com/vyasr) - Bradley Dice (https://github.com/bdice) - Karthikeyan (https://github.com/karthikeyann) - Nghia Truong (https://github.com/ttnghia) URL: https://github.com/rapidsai/cudf/pull/16385 commit 2eabe0de584ff8c8ae6e82b1845309d5b01c4a98 Merge: 4624edf586 8bba6dfad2 Author: gpuCI <[email protected]> Date: Wed Jul 24 18:16:08 2024 -0400 Merge pull request #16388 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 8bba6dfad239b4fd69a82acbc5dd7707ba576cce Author: Matthew Murray <[email protected]> Date: Wed Jul 24 18:16:03 2024 -0400 Migrate lists/set_operations to pylibcudf (#16190) Apart of #15162 Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Thomas Li (https://github.com/lithomas1) URL: https://github.com/rapidsai/cudf/pull/16190 commit 4624edf58683391529cd9d7b76ca2e45438655bf Merge: 077457ee89 73937fbaba Author: gpuCI <[email protected]> Date: Wed Jul 24 16:42:06 2024 -0400 Merge pull request #16387 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 73937fbabaeea76665663ed23688b1cac61b7ee9 Author: Matthew Murray <[email protected]> Date: Wed Jul 24 16:42:00 2024 -0400 Migrate lists/filling to pylibcudf (#16189) Apart of #15162 Authors: - Matthew Murray (https://github.com/Matt711) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Thomas Li (https://github.com/lithomas1) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16189 commit 077457ee89140e98c9e25849511b14410370f684 Merge: 17c1afbd93 8fcf72a787 Author: gpuCI <[email protected]> Date: Wed Jul 24 13:06:35 2024 -0400 Merge pull request #16382 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 8fcf72a787acb0168c97d11b8ab9130146e9b37e Author: Alessandro Bellina <[email protected]> Date: Wed Jul 24 12:06:29 2024 -0500 [JNI] Add setKernelPinnedCopyThreshold and setPinnedAllocationThreshold (#16288) In 24.08 two new cuDF methods are being added, and the second method is still in flight (see: https://github.com/rapidsai/cudf/pull/16206): ``` cudf::set_kernel_pinned_copy_threshold cudf::set_allocate_host_as_pinned_threshold ``` We'd like to expose these methods in our JNI layer. I created a Cudf.java with the two static methods, and put the definitions in CudfJni.cpp. Marked as draft since I need https://github.com/rapidsai/cudf/pull/16206 to merge, and we are still testing it. Authors: - Alessandro Bellina (https://github.com/abellina) - Nghia Truong (https://github.com/ttnghia) Approvers: - Robert (Bobby) Evans (https://github.com/revans2) - Jason Lowe (https://github.com/jlowe) URL: https://github.com/rapidsai/cudf/pull/16288 commit 17c1afbd936989bdcdcdb5654c1cbc4dbe57cc7d Merge: a0c58c766e 7191b74ce2 Author: gpuCI <[email protected]> Date: Wed Jul 24 09:55:53 2024 -0400 Merge pull request #16380 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 7191b74ce244518f17ef65e701f5a262f1c5cf8a Author: Matthew Roeschke <[email protected]> Date: Wed Jul 24 03:55:48 2024 -1000 Align Index __init__ APIs with pandas 2.x (#16362) * It would be nice to have `Index`'s constructor to not go through `IndexMeta.__call__`, but I think that would be a separate effort * There were a couple `verify_integrity` keyword arguments added that don't raise a `NotImplementedError` since there's not support, but I don't think it's worth making this case falling back in `cudf.pandas` as it's just a validation and won't affect further behavior with the object Authors: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16362 commit a0c58c766e41525059e5a4e37ac5fce3a638468e Merge: b66281c4fa 743264f6ac Author: gpuCI <[email protected]> Date: Wed Jul 24 06:32:36 2024 -0400 Merge pull request #16378 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 743264f6ac924fdbec58fad666f989b14b901a98 Author: brandon-b-miller <[email protected]> Date: Wed Jul 24 05:32:31 2024 -0500 Warn on cuDF failure when `POLARS_VERBOSE` is true (#16308) Just something quick to get us started here Closes https://github.com/rapidsai/cudf/issues/16256 Authors: - https://github.com/brandon-b-miller - Lawrence Mitchell (https://github.com/wence-) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/16308 commit b66281c4fa811431dec0cdc0d8222fba9e8e4088 Merge: f20205b2dc 62625f1bfc Author: gpuCI <[email protected]> Date: Wed Jul 24 03:42:08 2024 -0400 Merge pull request #16376 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 62625f1bfcdb980186a1afbec41e420fdb4a7075 Author: Matt Topol <[email protected]> Date: Wed Jul 24 03:42:03 2024 -0400 Host implementation of `to_arrow` using nanoarrow (#16297) Adds the corresponding `to_arrow_host` functions for interop using `ArrowDeviceArray`. This includes updating the version of nanoarrow in use to pick up some bug fixes and features. Authors: - Matt Topol (https://github.com/zeroshade) - Muhammad Haseeb (https://github.com/mhaseeb123) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Muhammad Haseeb (https://github.com/mhaseeb123) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16297 commit f20205b2dc7a5e830b72386df378934c53da5043 Merge: bc748d67b5 8c1749b40e Author: gpuCI <[email protected]> Date: Wed Jul 24 01:19:15 2024 -0400 Merge pull request #16375 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 8c1749b40eaa983966ed3bece6bdd29a4316d18a Author: Kyle Edwards <[email protected]> Date: Wed Jul 24 01:19:10 2024 -0400 Use rapids_cpm_bs_thread_pool() (#16360) Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) URL: https://github.com/rapidsai/cudf/pull/16360 commit bc748d67b52de4cf1c876f9701644fdbf1d839e5 Merge: 6d9aff4b7d 75289c58f3 Author: gpuCI <[email protected]> Date: Wed Jul 24 00:46:03 2024 -0400 Merge pull request #16374 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 75289c58f3d9ca11a51396e4adadfbd5f51856f5 Author: Bradley Dice <[email protected]> Date: Tue Jul 23 23:45:59 2024 -0500 Rename PrefetchConfig to prefetch_config. (#16358) This PR addresses a comment requesting a rename of `PrefetchConfig` to `prefetch_config`. See: https://github.com/rapidsai/cudf/pull/16020#discussion_r1686284151 Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Shruti Shivakumar (https://github.com/shrshi) - Nghia Truong (https://github.com/ttnghia) URL: https://github.com/rapidsai/cudf/pull/16358 commit 6d9aff4b7dfd23db43d294dacdeaf6c52af2fc4b Merge: dcf791c83e f0efc8b36a Author: gpuCI <[email protected]> Date: Tue Jul 23 20:17:10 2024 -0400 Merge pull request #16373 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit f0efc8b36a8f43cfa027966265dcea052bb5c45d Author: Vukasin Milovanovic <[email protected]> Date: Tue Jul 23 17:17:05 2024 -0700 Modify `make_host_vector` and `make_device_uvector` factories to optionally use pinned memory and kernel copy (#16206) Issue #15616 Modified `make_host_vector` functions to return `cudf::detail::host_vector`, which can use a pinned or a pageable memory resource. When pinned memory is used, the D2H copy is potentially done using a CUDA kernel. Also added factories to create `host_vector`s without device data. These are useful to replace uses of `std::vector` and `thrust::host_vector` when the data eventually gets copied to the GPU. Added `is_device_accessible` to `host_span`. With this, `make_device_uvector` can optionally use the kernel for the H2D copy. Modified `cudf::detail::host_vector` to be derived from `thrust::host_vector`, to avoid issues with implicit conversion from `std::vector`. Used `cudf::detail::host_vector` and its new factory functions wherever data ends up copied to the GPU. Stopped using `thrust::copy_n` for the kernel copy path in `cuda_memcpy` because of an optimization that allows it to fall back to `cudaMemCpyAsync`. We now call a simple local kernel. Authors: - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Robert Maynard (https://github.com/robertmaynard) - Yunsong Wang (https://github.com/PointKernel) - Nghia Truong (https://github.com/ttnghia) - Alessandro Bellina (https://github.com/abellina) URL: https://github.com/rapidsai/cudf/pull/16206 commit dcf791c83e3ab87d57d94017ee7413d96f9e99a5 Merge: 7a09f809dc 39f256c339 Author: gpuCI <[email protected]> Date: Tue Jul 23 20:03:22 2024 -0400 Merge pull request #16372 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 39f256c3397afc9c495cb819636abddb23f81dc0 Author: brandon-b-miller <[email protected]> Date: Tue Jul 23 19:03:16 2024 -0500 Fall back to CPU for unsupported libcudf binaryops in cudf-polars (#16188) This PR adds logic that should trigger CPU fallback unsupported binary ops. Authors: - https://github.com/brandon-b-miller - Lawrence Mitchell (https://github.com/wence-) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/16188 commit 7a09f809dc5c8cf8d2663fae186e4d249893c888 Merge: a3aacd8915 cd711913d2 Author: gpuCI <[email protected]> Date: Tue Jul 23 18:24:24 2024 -0400 Merge pull request #16370 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit cd711913d2312ba158e34f5c03784a7b07f1583a Author: Elias Stehle <[email protected]> Date: Wed Jul 24 00:24:19 2024 +0200 Adds write-coalescing code path optimization to FST (#16143) This PR adds an optimized code path to the finite-state transducer (FST) that will use a shared memory-backed write buffer for the translated output and translated output indexes, if the the write buffer does not require allocating excessive amounts of shared memory (i.e., current heuristic is 24 KB/CTA). Writes are first buffered in shared memory and then collaboratively written out using coalesced writes to global memory. ## Benchmark results Numbers are for libcudf's FST_NVBENCH for a 1.073 GB input. FST outputs one token per input symbol. Benchmarks run on V100 with 900 GB/s theoretical peak BW. We compare the current FST implementation (old) to an FST implementaation that uses write-coalescing to gmem (new). | | OLD throughput (GB/s) | NEW throughput (GB/s) | relative performance | | 1st kernel, per byte: bytes read/written | 2nd kernel, per byte: bytes read/written | expected SOL (GB/s) | achieved SOL (old) | achieved SOL (new) | |------------------|------------------------|------------------------|----------------------|---|------------------------------------------|------------------------------------------|---------------------|--------------------|--------------------| | full | 15.7 | 74.74 | 476% | | 1 | 6 | 102.86 | 15.26% | 72.66% | | no out-indexes | 39.123 | 105.8 | 270% | | 1 | 2 | 240.00 | 16.30% | 44.08% | | no-output | 229.27 | 178.92 | 78% | | 1 | 1 | 360.00 | 63.69% | 49.70% | | out-indexes-only | 24.95 | 85.2 | 341% | | 1 | 5 | 120.00 | 20.79% | 71.00% | Authors: - Elias Stehle (https://github.com/elstehle) Approvers: - Shruti Shivakumar (https://github.com/shrshi) - Vukasin Milovanovic (https://github.com/vuule) URL: https://github.com/rapidsai/cudf/pull/16143 commit a3aacd8915fa503ea4be8e1d7797a080e0427923 Merge: 2de9fa7bd8 ff30c02111 Author: gpuCI <[email protected]> Date: Tue Jul 23 15:04:01 2024 -0400 Merge pull request #16366 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit ff30c0211109e14b1f6918fcc6c2e2b98f863a1f Author: Nghia Truong <[email protected]> Date: Tue Jul 23 12:03:55 2024 -0700 Fix compile warnings with `jni_utils.hpp` (#16336) This fixes the compiler warnings with `jni_utils.hpp`, removing some `const` qualifiers that are redundant. Closes https://github.com/rapidsai/cudf/issues/16335. Authors: - Nghia Truong (https://github.com/ttnghia) Approvers: - Jason Lowe (https://github.com/jlowe) URL: https://github.com/rapidsai/cudf/pull/16336 commit 2de9fa7bd821c7b1653340dfca4e6a1e9e216cc5 Merge: bc609fb648 e6d412cba7 Author: gpuCI <[email protected]> Date: Tue Jul 23 07:03:33 2024 -0400 Merge pull request #16364 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit e6d412cba7c23df7ee500c28257ed9281cea49b9 Author: brandon-b-miller <[email protected]> Date: Tue Jul 23 06:03:28 2024 -0500 Fall back when casting a timestamp to numeric in cudf-polars (#16232) This PR adds logic that falls back to CPU when a cudf-polars query would cast a timestamp column to a numeric type, an unsupported operation in libcudf, which should fix a few polars tests. It could be cleaned up a bit with some of the utilities that will be added in https://github.com/rapidsai/cudf/pull/16150. Authors: - https://github.com/brandon-b-miller Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/16232 commit bc609fb6482e32152d64f3e9d34aaa4cb9b87cec Merge: 023dba6fab c7b28ceeb4 Author: gpuCI <[email protected]> Date: Tue Jul 23 06:28:20 2024 -0400 Merge pull request #16363 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit c7b28ceeb46d2b921e30f081a9ed97745c91ff9e Author: brandon-b-miller <[email protected]> Date: Tue Jul 23 05:28:13 2024 -0500 Add `drop_nulls` in `cudf-polars` (#16290) Closes https://github.com/rapidsai/cudf/issues/16219 Authors: - https://github.com/brandon-b-miller Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/16290 commit 023dba6fab1c00116b11ff10fc7536d4f9e78fcd Merge: 4a0813b681 0cac2a9d68 Author: gpuCI <[email protected]> Date: Mon Jul 22 17:18:26 2024 -0400 Merge pull request #16359 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 0cac2a9d68341a38721be16132ead14cf4a0d70b Author: Shruti Shivakumar <[email protected]> Date: Mon Jul 22 14:18:21 2024 -0700 Remove size constraints on source files in batched JSON reading (#16162) Addresses https://github.com/rapidsai/cudf/issues/16138 The batched multi-source JSON reader fails when the size of any of the input source buffers exceeds `INT_MAX` bytes. The goal of this PR is to remove this constraint by modifying the batching behavior of the reader. Instead of constructing batches that include entire source files, the batches are now constructed at the granularity of byte ranges of size at most `INT_MAX` bytes, Authors: - Shruti Shivakumar (https://github.com/shrshi) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - Karthikeyan (https://github.com/karthikeyann) URL: https://github.com/rapidsai/cudf/pull/16162 commit 4a0813b68158474b00d3e7c692310b62b48fe2fc Merge: a4acaa7177 81e65ee312 Author: gpuCI <[email protected]> Date: Mon Jul 22 16:18:45 2024 -0400 Merge pull request #16357 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 81e65ee312af5133ca2b98d52efaeb29c274a825 Author: GALI PREM SAGAR <[email protected]> Date: Mon Jul 22 15:18:40 2024 -0500 Fix docstring of `DataFrame.apply` (#16351) This PR fixes docstring of `DataFrame.apply` Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: https://github.com/rapidsai/cudf/pull/16351 commit a4acaa717798a3a09a57ab333965c00666d9d808 Merge: 0868314b1d 996cb8d870 Author: gpuCI <[email protected]> Date: Mon Jul 22 16:15:22 2024 -0400 Merge pull request #16356 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 996cb8d870b7b6153802bde670435e8cd3b8775d Author: Matthew Murray <[email protected]> Date: Mon Jul 22 16:15:16 2024 -0400 Migrate lists/sorting to pylibcudf (#16179) Apart of #15162 Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16179 commit 0868314b1d5f2ca31eb56f4fee5f75de42b22fbe Merge: a3ebf3badd c14c8bf59f Author: gpuCI <[email protected]> Date: Mon Jul 22 15:04:01 2024 -0400 Merge pull request #16355 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit c14c8bf59fd1e97fe94c8dfd2db6df7f9a6c65ad Author: Thomas Li <[email protected]> Date: Mon Jul 22 12:03:56 2024 -0700 Implement parquet reading using pylibcudf in cudf-polars (#16346) Replace cudf-classic with pylibcudf for parquet reading in cudf-polars. Authors: - Thomas Li (https://github.com/lithomas1) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16346 commit a3ebf3badd0c7375b3f24dd466d4db8fa127000e Merge: edbb1bcd9c e0a00c1fcb Author: gpuCI <[email protected]> Date: Mon Jul 22 15:03:29 2024 -0400 Merge pull request #16354 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit e0a00c1fcb4b72b7abd29debe5b2f6b38081d39a Author: Jayjeet Chakraborty <[email protected]> Date: Mon Jul 22 12:03:24 2024 -0700 Add `stream` param to list explode APIs (#16317) Add `stream` param to list `explode*` APIs. Partially fixes https://github.com/rapidsai/cudf/issues/13744 Authors: - Jayjeet Chakraborty (https://github.com/JayjeetAtGithub) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16317 commit edbb1bcd9c363876b79039caf7176270ee3eba03 Merge: b52ec0f436 e54b82c9f3 Author: gpuCI <[email protected]> Date: Mon Jul 22 15:03:09 2024 -0400 Merge pull request #16353 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit e54b82c9f3499b35e7e789d41d2042a5d5a80810 Author: Mark Harris <[email protected]> Date: Tue Jul 23 05:03:04 2024 +1000 Use resource_ref for upstream in stream_checking_resource_adaptor (#16187) As we move toward replacing all `device_memory_resource` pointers with `resource_ref`s, there are some places that changes can be made ahead of RMM to simplify required changes as RMM is refactored. In this PR I eliminate the unnecessary `Upstream` template parameter from `cudf_test::stream_checking_resource_adaptor`, and use a `device_async_resource` for the upstream resource. A similar change will be made to all RMM resource adaptors, but this one can be done without deprecations since it is just a test utility. Authors: - Mark Harris (https://github.com/harrism) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16187 commit b52ec0f436c549b79daf6d9379ad2851b8833dbe Merge: 0135e46880 3053f42351 Author: gpuCI <[email protected]> Date: Mon Jul 22 13:56:45 2024 -0400 Merge pull request #16352 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 3053f42351b04e22d873f78f5bc49f8b20ff17ac Author: Jayjeet Chakraborty <[email protected]> Date: Mon Jul 22 10:56:39 2024 -0700 Add missing `stream` param to dictionary factory APIs (#16319) Add `stream` param to dictionary column factory functions. Partially solves #13744 Authors: - Jayjeet Chakraborty (https://github.com/JayjeetAtGithub) Approvers: - Mark Harris (https://github.com/harrism) - Yunsong Wang (https://github.com/PointKernel) URL: https://github.com/rapidsai/cudf/pull/16319 commit 0135e468808ccf7e8471e654bcd723eafb9c48c5 Merge: c53f9c54ac 135c99512e Author: gpuCI <[email protected]> Date: Mon Jul 22 10:13:37 2024 -0400 Merge pull request #16344 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 135c99512e5f7a2d38f6a870ad6883ccb39a3cce Author: Matthew Roeschke <[email protected]> Date: Mon Jul 22 04:13:32 2024 -1000 Align Series APIs with pandas 2.x (#16333) Similar to https://github.com/rapidsai/cudf/pull/16310, the follow APIs have been modified to adjust/add parameters * `reindex` * `reset_index` * `add_suffix` * `searchsorted` * `clip` * `mask` * `shift` * `dropna` * `rename` * `cov` * `apply` * `replace` Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16333 commit c53f9c54ac9e4d25350f04ffcb41ceb5bca9bdb2 Merge: c636778de3 852b151002 Author: gpuCI <[email protected]> Date: Mon Jul 22 09:48:23 2024 -0400 Merge pull request #16343 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 852b151002dc76e9f09d3529c80e4b589f1df9fc Author: Lawrence Mitchell <[email protected]> Date: Mon Jul 22 14:48:18 2024 +0100 Fix issue in horizontal concat implementation in cudf-polars (#16271) Shorter tables must be extended to the same length as the longest table. Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16271 commit c636778de39491e24ace55d99dcfb29c574a20d2 Merge: dacc6c0baa e6537de747 Author: gpuCI <[email protected]> Date: Fri Jul 19 23:10:44 2024 -0400 Merge pull request #16342 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit e6537de7474c91b4153542e6611c8a4e33a58caa Author: Vyas Ramasubramani <[email protected]> Date: Fri Jul 19 20:10:40 2024 -0700 Experimental support for configurable prefetching (#16020) This PR adds experimental support for prefetching managed memory at a select few points in libcudf. A new configuration object is introduced for handling whether prefetching is enabled or disabled, and whether to print debug information about pointers being prefetched. Prefetching control is managed on a per API basis to enable profiling of the effects of prefetching different classes of data in different contexts. Prefetching in this PR always occurs on the default stream, so it will trigger synchronization with any blocking streams that the user has created. Turning on prefetching and then passing non-blocking to any libcudf APIs will trigger undefined behavior. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - David Wendt (https://github.com/davidwendt) - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Thomas Li (https://github.com/lithomas1) - Muhammad Haseeb (https://github.com/mhaseeb123) URL: https://github.com/rapidsai/cudf/pull/16020 commit dacc6c0baa47c89fe8e0d1c3d246bcc94a4b6416 Merge: 1ccdf15dd7 c5b96003ce Author: gpuCI <[email protected]> Date: Fri Jul 19 23:04:24 2024 -0400 Merge pull request #16341 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit c5b96003cef00b2635923d03edcd48a13821a61e Author: Thomas Li <[email protected]> Date: Fri Jul 19 20:04:19 2024 -0700 Migrate Parquet reader to pylibcudf (#16078) xref #15162 Migrates the parquet reader (and chunked parquet reader) to pylibcudf. (Does not migrate the writers or the metadata reader yet). Authors: - Thomas Li (https://github.com/lithomas1) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/16078 commit 1ccdf15dd736a1a08aa8f566a47ca0392ca33cac Merge: 97e1bab151 26a3799d2f Author: gpuCI <[email protected]> Date: Fri Jul 19 22:49:07 2024 -0400 Merge pull request #16340 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 26a3799d2ff9ffb2aa72d63bb388b4bee70b3440 Author: Matthew Roeschke <[email protected]> Date: Fri Jul 19 16:49:01 2024 -1000 Make ColumnAccessor strictly require a mapping of columns (#16285) `ColumnAccessor` had a default `data=None` argument and initialized an empty dict in the `__init__` if `data` was not passed. This PR now makes `data` a required argument. Additionally if `verify=True`, the `__init__` would call `as_column` on each `data.values()` allowing non-`ColumnBase` inputs. This PR now avoids this call and makes the caller responsible for ensuring the inputs are `ColumnBase`s Also, adds a few `verify=False` internally where we know we are passing columns from a libcudf op or reconstructing from another `ColumnAccessor` Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16285 commit 97e1bab151184aa537edf39b7e838c07e07271a9 Merge: 5ad4c877ed 75335f6af5 Author: gpuCI <[email protected]> Date: Fri Jul 19 21:21:32 2024 -0400 Merge pull request #16339 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 75335f6af51bde6be68c1fb0a6caa8030b9eda3e Author: Muhammad Haseeb <[email protected]> Date: Fri Jul 19 18:21:27 2024 -0700 Report number of rows per file read by PQ reader when no row selection and fix segfault in chunked PQ reader when skip_rows > 0 (#16195) Closes #15389 Closes #16186 This PR adds the capability to calculate and report the number of rows read from each data source into the table returned by the Parquet reader (both chunked and normal). The returned vector of counts is only valid (non-empty) when row selection (AST filter) is not being used. This PR also fixes a segfault in chunked parquet reader when skip_rows > 0 and the number of passes > 1. This segfault was being caused by a couple of arithmetic errors when computing the (start_row, num_row) for row_group_info, pass, column chunk descriptor structs. Both changes were added to this PR as changes and the gtests from the former work were needed to implement the segfault fix. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - Vukasin Milovanovic (https://github.com/vuule) URL: https://github.com/rapidsai/cudf/pull/16195 commit 5ad4c877ed631094f358f87c003ee9b381e9e270 Merge: ebacf394d9 535db9b26e Author: gpuCI <[email protected]> Date: Fri Jul 19 20:28:20 2024 -0400 Merge pull request #16338 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 535db9b26ed1a57e4275f4a6f11b04ebeee21248 Author: Thomas Li <[email protected]> Date: Fri Jul 19 17:28:14 2024 -0700 Deprecate Arrow support in I/O (#16132) Contributes to https://github.com/rapidsai/cudf/issues/15193 Authors: - Thomas Li (https://github.com/lithomas1) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Richard (Rick) Zamora (https://github.com/rjzamora) - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/16132 commit ebacf394d975fa5a0f65a7337d5587c9e8273902 Merge: b11cdf854d e169e8e427 Author: gpuCI <[email protected]> Date: Fri Jul 19 19:36:08 2024 -0400 Merge pull request #16337 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit e169e8e4273e4d317e3f27c810c5b137dd75adb3 Author: Thomas Li <[email protected]> Date: Fri Jul 19 16:36:03 2024 -0700 Implement read_csv in cudf-polars using pylibcudf (#16307) Replace cudf-classic with pylibcudf for CSV reading in cudf-polars Authors: - Thomas Li (https://github.com/lithomas1) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Lawrence Mitchell (https://github.com/wence-) URL: https://github.com/rapidsai/cudf/pull/16307 commit b11cdf854d64e248d682ad2d8178f8ae08e34b3e Merge: d82caec4e0 5dde41d7f7 Author: gpuCI <[email protected]> Date: Fri Jul 19 19:08:41 2024 -0400 Merge pull request #16334 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 5dde41d7f7533180ecd355bac248a7ed18adcc10 Author: Matthew Roeschke <[email protected]> Date: Fri Jul 19 13:08:36 2024 -1000 Replace is_float/integer_dtype checks with .kind checks (#16261) It appears this was called when we already had a dtype object so can instead just simply check the .kind attribute Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16261 commit d82caec4e04468b497f2d553221c6314c53f9d10 Merge: 3c3ee56637 4c46628eaf Author: gpuCI <[email protected]> Date: Fri Jul 19 18:51:12 2024 -0400 Merge pull request #16332 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 4c46628eaf7ba16a2a181ceb3311f315cd4932dc Author: Matthew Roeschke <[email protected]> Date: Fri Jul 19 12:51:07 2024 -1000 Mark cudf._typing as a typing module in ruff (#16318) Additionally breaks up the prior, single-line of `select` rules that are enabled. Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Thomas Li (https://github.com/lithomas1) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16318 commit 3c3ee56637116e07804f20efab46d4dd3aa7c4cf Merge: 1cb07e0c29 7d3083254c Author: gpuCI <[email protected]> Date: Fri Jul 19 18:48:43 2024 -0400 Merge pull request #16331 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 7d3083254c0503b07f82af32188120f42acef860 Author: Matthew Roeschke <[email protected]> Date: Fri Jul 19 12:48:39 2024 -1000 Replace np.isscalar/issubdtype checks with is_scalar/.kind checks (#16275) * `is_scalar` also handles cudf.Scalars which should be handled internally * `issubdtype` can largely be replaced by checking the `.kind` attribute on the dtype Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16275 commit 1cb07e0c29c0b6acd1896ecef867afeca27a84c1 Merge: 52657b3375 57ed7fce67 Author: gpuCI <[email protected]> Date: Fri Jul 19 18:25:01 2024 -0400 Merge pull request #16330 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 57ed7fce6742abc96a8fd65216f032bad5937a2f Author: brandon-b-miller <[email protected]> Date: Fri Jul 19 17:24:55 2024 -0500 Add tests for `pylibcudf` binaryops (#15470) This PR implements a more general approach to testing binaryops that originally came up in https://github.com/rapidsai/cudf/pull/15279. This PR can possibly supersede that one. Authors: - https://github.com/brandon-b-miller Approvers: - Lawrence Mitchell (https://github.com/wence-) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/15470 commit 52657b3375c900a66b6ec5f8d7e1ebe37c38232f Merge: 6be515506d ecc27a1140 Author: gpuCI <[email protected]> Date: Fri Jul 19 17:55:45 2024 -0400 Merge pull request #16329 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit ecc27a1140c0c287091f6a1291dfaf7ccd82cb19 Author: Matthew Roeschke <[email protected]> Date: Fri Jul 19 11:55:40 2024 -1000 Align more DataFrame APIs with pandas (#16310) I have a script that did some signature comparisons between `pandas.DataFrame` and `cudf.DataFrame` API and it appears some signatures have changed between the pandas 1.x and 2.x release. The API changes in this PR are mostly adding implementations or adding missing keyword argument (although they might not be implemented). The APIs affected are: * `__init__` * `__array__` * `__arrow_c_stream__` * `to_dict` * `where` * `add_prefix` * `join` * `apply` * `to_records` * `from_records` * `unstack` * `pct_change` * `sort_values` Marking as breaking as I ensured some added keywords are in the same positions as pandas and therefore might break users who are using purely positional arguments. Authors: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16310 commit 6be515506d4a6f833e71ac67f16c2925f7b8576b Merge: fcaea56166 6e37afc7c9 Author: gpuCI <[email protected]> Date: Fri Jul 19 17:52:32 2024 -0400 Merge pull request #16328 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 6e37afc7c9e177b307c41950e52453bd5906af44 Author: Matthew Roeschke <[email protected]> Date: Fri Jul 19 11:52:27 2024 -1000 Make __bool__ raise for more cudf objects (#16311) To match pandas, this PR makes `DataFrame`, `MultiIndex` and `RangeIndex` raise on `__bool__`. Authors: - Matthew Roeschke (https://github.com/mroeschke) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: https://github.com/rapidsai/cudf/pull/16311 commit fcaea56166e2d8f8b1916d702ec8572a9e12b2be Merge: 051fadd250 910989eb8f Author: gpuCI <[email protected]> Date: Fri Jul 19 17:48:42 2024 -0400 Merge pull request #16327 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 910989eb8fb87b2e896aa032260705c27cce71e0 Author: Bradley Dice <[email protected]> Date: Fri Jul 19 15:48:37 2024 -0600 Rename gather/scatter benchmarks to clarify coalesced behavior. (#16083) The benchmark names `coalesce_x` and `coalesce_o` are not very clear. This PR renames them to `coalesced` and `shuffled`. This was discussed with @GregoryKimball. Authors: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Karthikeyan (https://github.com/karthikeyann) - Mike Wilson (https://github.com/hyperbolic2346) URL: https://github.com/rapidsai/cudf/pull/16083 commit 051fadd2500bc20b90b74d662deec918ee27f299 Merge: ece86996ad fa0d89d9b4 Author: gpuCI <[email protected]> Date: Fri Jul 19 17:46:33 2024 -0400 Merge pull request #16326 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit fa0d89d9b4b4152b919999b5f01b1e68407469c5 Author: Matthew Roeschke <[email protected]> Date: Fri Jul 19 11:46:28 2024 -1000 Clean unneeded/redudant dtype utils (#16309) * Replace `min_scalar_type` with `min_signed_type` (the former just called the latter) * Replace `numeric_normalize_types` with `find_common_dtype` followed by a column `astype` * Removed `_NUMPY_SCTYPES` with just hardcoding the integer/floating types or using `np.integer`/`np.floating` Authors: - Matthew Roeschke (https://github.com/mroeschke) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16309 commit ece86996ad69b1631e0da6f4dfb551cda38585a8 Merge: f47c891a2e 18f5fe0010 Author: gpuCI <[email protected]> Date: Fri Jul 19 17:41:47 2024 -0400 Merge pull request #16325 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 18f5fe0010fd42f604a340cd025a9ca9e122c6f5 Author: Thomas Li <[email protected]> Date: Fri Jul 19 14:41:39 2024 -0700 Fix polars for 1.2.1 (#16316) I think Polars made a breaking change in a patch release. At least the error we're getting looks like the error from https://github.com/pola-rs/polars/pull/17606. Authors: - Thomas Li (https://github.com/lithomas1) Approvers: - Lawrence Mitchell (https://github.com/wence-) - Vyas Ramasubramani (https://github.com/vyasr) URL: https://github.com/rapidsai/cudf/pull/16316 commit f47c891a2ea3a0de4bb0462d557531e046860fbb Merge: c61638cbeb 3df4ac2842 Author: gpuCI <[email protected]> Date: Fri Jul 19 16:46:23 2024 -0400 Merge pull request #16323 from rapidsai/branch-24.08 Forward-merge branch-24.08 into branch-24.10 commit 3df4ac28423b99e4dd88570da8d55e2e5af2e1bc Author: Matthew Roeschke <[email protected]> Date: Fri Jul 19 10…

sisodia1701 · 2024-08-07T16:38:58Z

Waiting for the tests to pass on the RAPIDS 24.10 build.
Things related to rapids build backend are looking good, hence closing this issue.

Contributes to rapidsai/build-planning#31. Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Jake Awe (https://github.com/AyodeAwe) - Dante Gama Dessavre (https://github.com/dantegd) - James Lamb (https://github.com/jameslamb) - Bradley Dice (https://github.com/bdice) URL: rapidsai/cuvs#145

…nsolidate some dependency lists (#16611) Proposes some additional cleanup in `dependencies.yaml`, for things I noticed while working through #15483. * standardizes the naming of keys in the `files:` section for build dependencies - *`py_build_{project}` = dependencies for the `[build-system]` table* - *`py_rapids_build_{project}` = dependencies for the `[tool.rapids-build-backend]` table* - *this is how it was done over most of the other repos in rapidsai/build-planning#31, it was just missed because `cudf` was one of the first repos to add `rapids-build-backend`* * removes the dependency on building `streamz` from latest source on GitHub - *`custreamz` conda packages and wheels depend on packages for those, not this git dependency* - https://github.com/rapidsai/cudf/blob/2f7d35435db2b5ed9ead96cf43e2a710db5e5e6d/dependencies.yaml#L752-L754 - https://github.com/rapidsai/cudf/blob/2f7d35435db2b5ed9ead96cf43e2a710db5e5e6d/conda/recipes/custreamz/meta.yaml#L45-L47 - *if this is really needed, I don't think it belongs in the `build_python_cudf` set* - *the last commit to `streamz` was 2 years ago (https://github.com/python-streamz/streamz), this doesn't seem like a `rapids-dask-dependency`, try-to-always-test-against-latest, situation to me* - *I'm guessing this is left over from a time before `streamz` was regularly publishing wheels... it's been in `dependencies.yaml` since that file was first introduced here in November 2022 (#11674)* - *the last release, v0.6.4, was made on July 27, 2022. There have been around 20 commits to `master` since then ([history link](https://github.com/python-streamz/streamz/commits/master/)) ... but if `custreamz` really needed those, I'd expect `custreamz` to depend on the version built from GitHub sources. I strongly suspect that that isn't the case.* * removes `build_python_cudf` and `build_python_libcudf` lists in `dependencies.yaml`, in favor of re-using the `depends_on_rmm` and `depends_on_pylibcudf` lists Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #16611

This was referenced Mar 13, 2024

Use rapids-build-backend. rapidsai/rmm#1502

Merged

Use rapids-build-backend. rapidsai/cudf#15245

Merged

vyasr mentioned this issue Apr 6, 2024

Use CUDA wheels to avoid statically linking CUDA components in our wheels #35

Closed

vyasr mentioned this issue Apr 23, 2024

Simplify and standardize (or replace) build.sh scripts #53

Open

jameslamb mentioned this issue Apr 25, 2024

Use rapids-build-backend rapidsai/cuml#5804

Merged

jameslamb mentioned this issue May 2, 2024

add mypy to linting checks rapidsai/rapids-build-backend#19

Merged

jameslamb assigned KyleFromNVIDIA and jameslamb May 3, 2024

This was referenced May 3, 2024

Treat clobber warnings as errors for conda builds #54

Open

remove require_cuda defaults in internal functions rapidsai/rapids-build-backend#23

Merged

KyleFromNVIDIA mentioned this issue May 7, 2024

Use rapids-build-backend rapidsai/cugraph#4393

Merged

jakirkham added this to the v24.06.00 milestone May 15, 2024

jakirkham modified the milestones: v24.06.00, v24.08.00 May 15, 2024

jameslamb mentioned this issue May 21, 2024

Remove unused setuptools dependencies #62

Closed

6 tasks

This was referenced May 22, 2024

Use rapids-build-backend rapidsai/cuvs#145

Merged

Use rapids-build-backend rapidsai/raft#2331

Merged

sisodia1701 closed this as completed Aug 7, 2024

jameslamb mentioned this issue Aug 20, 2024

remove streamz git dependency, standardize build dependency names, consolidate some dependency lists rapidsai/cudf#16611

Merged

3 tasks

jameslamb mentioned this issue Aug 21, 2024

Use CUDA math wheels rapidsai/raft#2415

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update RAPIDS Python packages to use rapids-build-backend #31

Update RAPIDS Python packages to use rapids-build-backend #31

vyasr commented Mar 13, 2024 •

edited by jameslamb

Loading

Changes to `rapids-build-backend` and `rapids-dependency-file-generator`

Migrations

Other

dependencies.yaml updates (for devcontainers / DLFW)

mmccarty commented May 15, 2024

jakirkham commented May 15, 2024

jameslamb commented May 28, 2024

sisodia1701 commented Aug 7, 2024

Update RAPIDS Python packages to use rapids-build-backend #31

Update RAPIDS Python packages to use rapids-build-backend #31

Comments

vyasr commented Mar 13, 2024 • edited by jameslamb Loading

Changes to rapids-build-backend and rapids-dependency-file-generator

Migrations

Other

dependencies.yaml updates (for devcontainers / DLFW)

mmccarty commented May 15, 2024

jakirkham commented May 15, 2024

jameslamb commented May 28, 2024

sisodia1701 commented Aug 7, 2024

vyasr commented Mar 13, 2024 •

edited by jameslamb

Loading

Changes to `rapids-build-backend` and `rapids-dependency-file-generator`