Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] cuml v24.12 #6164

Merged
merged 37 commits into from
Dec 11, 2024
Merged

[RELEASE] cuml v24.12 #6164

merged 37 commits into from
Dec 11, 2024

Conversation

GPUtester
Copy link
Contributor

❄️ Code freeze for branch-24.12 and v24.12 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-24.12 until release (merging of this PR).

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-24.12 into main for the release

raydouglass and others added 30 commits September 19, 2024 12:05
@raydouglass
DOC v24.12 Updates [skip ci]
f8fddcd
@GPUtester
Merge pull request #6077 from rapidsai/branch-24.10
9ca2835 
Forward-merge branch-24.10 into branch-24.12
@jameslamb
Merge branch 24.10 into branch 24.12
746d3ff
@AyodeAwe
Merge pull request #6083 from jameslamb/branch-24.12-merge-24.10
117baba 
Merge branch 24.10 into branch 24.12
@GPUtester
Merge pull request #6086 from rapidsai/branch-24.10
260584a 
Forward-merge branch-24.10 into branch-24.12
@GPUtester
Merge pull request #6092 from rapidsai/branch-24.10
5db33c0 
Forward-merge branch-24.10 into branch-24.12
@GPUtester
Merge pull request #6095 from rapidsai/branch-24.10
0ff653b 
Forward-merge branch-24.10 into branch-24.12
@KyleFromNVIDIA
Prune workflows based on changed files (#6094)
4df6a48 
Contributes to rapidsai/build-planning#94

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #6094
@KyleFromNVIDIA
Remove temporary exclusion of .github/workflows/ (#6097)
c78a748 
Follow-up to #6094
@GPUtester
Merge pull request #6098 from rapidsai/branch-24.10
82483ac 
Forward-merge branch-24.10 into branch-24.12
@Matt711
Update all rmm imports to use pylibrmm/librmm (#6084)
61f85a6 
This PR updates all the RMM imports to use pylibrmm/librmm now that `rmm._lib` is deprecated . It should be merged after [rmm/1676](rapidsai/rmm#1676).

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6084
@jameslamb
make conda installs in CI stricter (#6103)
22342aa 
Contributes to rapidsai/build-planning#106

Proposes specifying the RAPIDS version in `conda install` calls that install CI artifacts, to reduce the risk of CI jobs picking up artifacts from other releases.

## Notes for Reviewers

This only changes the docs build, because other packages already solve in a single `conda install` with version constraints, thanks to #5781.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6103
@divyegala
Merge branch-24.10 into branch-24.12
ded22dd
@AyodeAwe
Merge pull request #6106 from divyegala/branch-24.12-merge-24.10
592168e 
Forward merge Branch 24.10 into 24.12
@jameslamb
fix version in Doxygen docs (#6104)
ef7a61d 
Follow-up to #6103.

In that PR, I'd removed the `export RAPIDS_MAJOR_MINOR_VERSION`. I realized that this project's Doxygen setup actually expects that to be set in the environment.

https://github.com/rapidsai/cuml/blob/22342aad0ecd327e3140471c479d01d83b932c23/ci/build_docs.sh#L35

https://github.com/rapidsai/cuml/blob/22342aad0ecd327e3140471c479d01d83b932c23/cpp/Doxyfile.in#L41

This fixes that, sorry 😬

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - https://github.com/jakirkham
  - Mike Sarahan (https://github.com/msarahan)

URL: #6104
@viclafargue
Fix Dask estimators serialization prior to training (#6065)
417d980 
Partially answers #6046

Authors:
  - Victor Lafargue (https://github.com/viclafargue)
  - Dante Gama Dessavre (https://github.com/dantegd)

Approvers:
  - Divye Gala (https://github.com/divyegala)

URL: #6065
@divyegala
Enable HDBSCAN gpu training and cpu inference (#6108)
fb4c8af 
Until now, we supported all combinations of GPU/CPU interoperability except the one mentioned in the title. This was because the CPU HDBSCAN package was missing attribute setters. With scikit-learn-contrib/hdbscan#657, attribute setters are now available which allow us to transfer GPU trained attributes to the CPU model. This feature is available as part of `hdbscan=0.8.39`

Authors:
  - Divye Gala (https://github.com/divyegala)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)
  - Bradley Dice (https://github.com/bdice)
  - Dante Gama Dessavre (https://github.com/dantegd)
  - William Hicks (https://github.com/wphicks)

URL: #6108
@wphicks
Remove type from pickle header for CumlArray (#6120)
addccbd 
Including the type in the header is unnecessary for (de)serialization

Authors:
  - William Hicks (https://github.com/wphicks)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6120
@jjacobelli
devcontainer: replace VAULT_HOST with AWS_ROLE_ARN (#6118)
a5e35c5 
This PR is replacing the `VAULT_HOST` variable with `AWS_ROLE_ARN`. This is required to use the new token service to get AWS credentials.

Authors:
  - Jordan Jacobelli (https://github.com/jjacobelli)

Approvers:
  - Paul Taylor (https://github.com/trxcllnt)
  - Bradley Dice (https://github.com/bdice)

URL: #6118
@robertmaynard
cuml SINGLEGPU now tells cuvs to not build with nccl/mg support (#6127)
2feb7d8 
Corrects compilation failures when `SINGLEGPU` is enabled on machines without NCCL.

Authors:
  - Robert Maynard (https://github.com/robertmaynard)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6127
@jameslamb
print sccache stats in builds (#6111)
27e47e5 
Contributes to rapidsai/build-planning#111

Proposes some small packaging/CI changes, matching similar changes being made across RAPIDS.

* printing `sccache` stats to CI logs
* reducing `pip`'s verbosity in wheel building scripts
* updating to the latest `rapids-dependency-file-generator` (v1.16.0)
* always explicitly specifying `cpp` / `python` in calls to `rapids-upload-wheels-to-s3`

## Notes for Reviewers

This originally also ran wheel builds with `--no-build-isolation`, but I reverted that based on rapidsai/build-planning#108 (comment).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #6111
@hcho3
Make cuVS optional if CUML_ALGORITHMS is set (#6125)
a0186c7 
cuVS should not be required when building cuML with `CUML_ALGORITHMS=fil`.

Authors:
  - Philip Hyunsu Cho (https://github.com/hcho3)

Approvers:
  - William Hicks (https://github.com/wphicks)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6125
@bdice
Put a ceiling on cuda-python (#6131)
9150279 
This project is incompatible with newer versions of `cuda-python`. This puts ceilings of `<=11.8.3` (CUDA 11) and `<=12.6.0` (CUDA 12) on that library.

Those ceilings should be removed and replaced with `!=` constraints once new releases of `cuda-python` are up that this project is compatible with.

See rapidsai/build-planning#116 for more information.

Authors:
  - Bradley Dice (https://github.com/bdice)
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #6131
@wphicks
Correctly handle missing categorical data in experimental FIL (#6132)
22c3ee8 
Correctly handle missing categorical data in FIL when 0 is one of the included categories.

Resolve #5578.

Authors:
  - William Hicks (https://github.com/wphicks)

Approvers:
  - Philip Hyunsu Cho (https://github.com/hcho3)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6132
@dantegd
Make get_param_names a class method on single GPU estimators to mat…
8e195fb 
…ch Scikit-learn closer (#6101)

Small difference between our estimators and Scikit-learn is that `get_param_names` are a classmethod in sklearn, and not in ours. This can make a few corner cases fail for using our estimators when Scikit-learn like estimators are expected. This PR fixes that. 

**Note:** This will not include dask-based estimators for the time being since they depend on introspection at object creation time.

Authors:
  - Dante Gama Dessavre (https://github.com/dantegd)
  - Divye Gala (https://github.com/divyegala)

Approvers:
  - William Hicks (https://github.com/wphicks)

URL: #6101
@robertmaynard
Don't presume pointers are mutually exclusive for device/host. (#6128)
56e5e62 
A pointer can be usable on the device and host at the same time. We can't invert `is_dev_ptr()` to check that something is a host pointer.

Here is the results of looking at the cudaPointerGetAttributes of different allocation types. As we can see things like cudaMallocManaged and cudaMallocHost allow the same pointer to be both host and device.
```
cudaPointerGetAttributes attributes malloc ptr
  is_dev_ptr  -> 0
  is_host_ptr -> 1
  memory loc  -> unregistered

cudaPointerGetAttributes attributes cudaMalloc ptr
  is_dev_ptr  -> 1
  is_host_ptr -> 0
  memory loc  -> device

cudaPointerGetAttributes attributes cudaMallocManaged cudaMemAttachGlobal ptr
  is_dev_ptr  -> 1
  is_host_ptr -> 1
  memory loc  -> managed

cudaPointerGetAttributes attributes cudaMallocManaged cudaMemAttachHost ptr
  is_dev_ptr  -> 1
  is_host_ptr -> 1
  memory loc  -> managed

cudaPointerGetAttributes attributes cudaMallocHost ptr
  is_dev_ptr  -> 1
  is_host_ptr -> 1
  memory loc  -> host
```

Authors:
  - Robert Maynard (https://github.com/robertmaynard)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)
  - William Hicks (https://github.com/wphicks)
  - Divye Gala (https://github.com/divyegala)

URL: #6128
@jameslamb
enforce wheel size limits, README formatting in CI (#6136)
8711e44 
Contributes to rapidsai/build-planning#110

Proposes adding 2 types of validation on wheels in CI, to ensure we continue to produce wheels that are suitable for PyPI.

* checks on wheel size (compressed),
  - *to be sure they're under PyPI limits*
  - *and to prompt discussion on PRs that significantly increase wheel sizes*
* checks on README formatting
  - *to ensure they'll render properly as the PyPI project homepages*
  - *e.g. like how https://github.com/scikit-learn/scikit-learn/blob/main/README.rst becomes https://pypi.org/project/scikit-learn/*

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #6136
@msarahan
add telemetry (#6126)
0f32e99 
Enables telemetry during cuml's build process. This is currently done by parsing Github Actions run log metadata, and should have no impact on build/test times 

Implement OpenTelemetry, as described in rapidsai/build-infra#139

Authors:
  - Mike Sarahan (https://github.com/msarahan)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #6126
@dantegd @divyegala
Experimental command line interface UX (#6135)
06958c4 
PR adds a first version of a command line user experience that covers
the following estimators:

- Linear Regression, Ridge, Lasso and ElastiNet
- Logistic Regression
- PCA and tSVD
- DBSCAN, KMeans and HDBSCAN
- UMAP and TSNE
- Nearest Neighbors

---------

Co-authored-by: divyegala <[email protected]>
@vyasr
Stop excluding cutlass from symbol exclusion check (#6140)
4fcdeb1 
Depends on rapidsai/raft#2503, which includes the kernel visibility fixes needed from cutlass.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #6140
benfred and others added 5 commits December 2, 2024 22:59
@benfred
Use sparse knn / distances from cuvs (#6143)
88b81b0 
Use the sparse knn and distances from cuvs instead of from raft.

This also allows us to switch over to the cuvs DistanceType, instead of the raft DistanceType (which is now only needed for the RBC code)

Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #6143
@hcho3
Update FIL tests to use XGBoost UBJSON instead of binary (#6153)
53259f5 
Starting from 2.1, XGBoost uses UBJSON format to serialize models.
Replace all uses of the legacy model format with UBJSON.

Also make `xgboost` a test dependency of cuML so that the FIL tests run in the CI pipelines.

Authors:
  - Philip Hyunsu Cho (https://github.com/hcho3)

Approvers:
  - William Hicks (https://github.com/wphicks)
  - Robert Maynard (https://github.com/robertmaynard)
  - https://github.com/jakirkham

URL: #6153
@trxcllnt
Fix scikit-learn version specifier (#6158)
7e6dbc0 
This version should be `1.5.*`, otherwise this causes pip solves to fail with the following error when included with a wheel downloaded via `pip download`:

```
ERROR: Cannot install scikit-learn, scikit-learn 1.5.2 (from scikit_learn-1.5.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl) and scikit-learn==1.5 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested scikit-learn 1.5.2 (from scikit_learn-1.5.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl)
    The user requested scikit-learn
    The user requested scikit-learn==1.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict
```

Authors:
  - Paul Taylor (https://github.com/trxcllnt)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6158
@lijinf2
Optimize MG variance calculation for dataset standardization for logi…
de96f3a 
…stic regression (#6138)

MG variance calculation currently involks raft SG vars API. However, the abs() step of raft SG vars API introduces errors in skewed data distribution (e.g., one GPU gets small values 1 and 2, and the other GPU gets large values 98 and 99). 

The PR avoids the effect of abs() when involking SG vars for calculating MG vars. The key idea is to pass a vector of zeroes when calling SG vars.

Authors:
  - Jinfeng Li (https://github.com/lijinf2)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6138
@lijinf2
Ensure MG to have the same number of allreduce calls in mean_stddev f…
4bfe72f 
…or sparse matrix to avoid hanging (#6141)

The hanging occurs when one GPU gets a sparse matrix of all zero values, while other GPUs get-zero values.

Authors:
  - Jinfeng Li (https://github.com/lijinf2)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #6141
@GPUtester GPUtester requested review from a team as code owners December 6, 2024 19:11
@GPUtester GPUtester requested review from KyleFromNVIDIA, vyasr, csadorf, divyegala and teju85 and removed request for a team December 6, 2024 19:11
@github-actions github-actions bot added conda conda issue Cython / Python Cython or Python issue CMake CUDA/C++ ci labels Dec 6, 2024
raydouglass and others added 2 commits December 11, 2024 13:11
@raydouglass
Update Changelog [skip ci]
70c04d9
@bdice
Fix SSL error. (#6177)
6dfc61b 
Last night, [wheel-tests-cuml / 11.8.0, 3.10, arm64, rockylinux8, a100,
latest-driver,
oldest-deps](https://github.com/rapidsai/cuml/actions/runs/12273029767/job/34243006376#logs)
failed.

The failure occurred in `testing/dask/utils.py`'s `load_text_corpus`:


https://github.com/rapidsai/cuml/blob/052cddef9648c3266974a0b43970afb347ba9d01/python/cuml/cuml/testing/dask/utils.py#L12

The errors looked like:

```
FAILED test_dask_naive_bayes.py::test_basic_fit_predict - urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>
FAILED test_dask_naive_bayes.py::test_single_distributed_exact_results - urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>
FAILED test_dask_naive_bayes.py::test_score - urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>
```

This adopts some fixes from
rapidsai/cugraph#4825 that will hopefully help
with SSL certificate failures.
@raydouglass raydouglass merged commit 41b1212 into main Dec 11, 2024
95 of 99 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KyleFromNVIDIA KyleFromNVIDIA Awaiting requested review from KyleFromNVIDIA KyleFromNVIDIA is a code owner automatically assigned from rapidsai/packaging-codeowners

@vyasr vyasr Awaiting requested review from vyasr vyasr is a code owner automatically assigned from rapidsai/cuml-cmake-codeowners

@csadorf csadorf Awaiting requested review from csadorf csadorf is a code owner automatically assigned from rapidsai/cuml-python-codeowners

@divyegala divyegala Awaiting requested review from divyegala divyegala is a code owner automatically assigned from rapidsai/cuml-python-codeowners

@teju85 teju85 Awaiting requested review from teju85 teju85 is a code owner automatically assigned from rapidsai/cuml-cpp-codeowners

Assignees
No one assigned
Labels
ci CMake conda conda issue CUDA/C++ Cython / Python Cython or Python issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.