Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make conda installs in CI stricter #17013

Merged
merged 5 commits into from
Oct 8, 2024
Merged

make conda installs in CI stricter #17013

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e6207d1
make package installations in CI stricter
jameslamb Oct 8, 2024
90a3979
revert the cudf-polars changes (polars needs to be installed first)
jameslamb Oct 8, 2024
278ecec
Merge branch 'branch-24.12' of github.com:rapidsai/cudf into ci-conda…
jameslamb Oct 8, 2024
911e9e4
Merge branch 'branch-24.12' of github.com:rapidsai/cudf into ci-conda…
jameslamb Oct 8, 2024
16367a5
clean up variables
jameslamb Oct 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions ci/build_docs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@

set -euo pipefail

export RAPIDS_VERSION="$(rapids-version)"
export RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"
RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"
export RAPIDS_VERSION_NUMBER="$RAPIDS_VERSION_MAJOR_MINOR"

rapids-logger "Create test conda environment"
Expand All @@ -29,7 +28,10 @@ PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)
rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
libcudf pylibcudf cudf dask-cudf
"libcudf=${RAPIDS_VERSION_MAJOR_MINOR}" \
"pylibcudf=${RAPIDS_VERSION_MAJOR_MINOR}" \
"cudf=${RAPIDS_VERSION_MAJOR_MINOR}" \
"dask-cudf=${RAPIDS_VERSION_MAJOR_MINOR}"

export RAPIDS_DOCS_DIR="$(mktemp -d)"

Expand Down
7 changes: 6 additions & 1 deletion ci/test_cpp_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"
Copy link
Contributor

@bdice bdice Oct 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In build_docs.sh we export this, but here we're just setting it locally. I doubt it matters here but it might be worth establishing consistent patterns. Also applies to other files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking here was "export it if it's already exported (as in the docs example), otherwise make it local", since I thought that had the least risk of causing other surprising problems.

I clicked around a little, and I think:

  • RAPIDS_VERSION = does not even need to be set in docs-build scripts
  • RAPIDS_VERSION_MAJOR_MINOR = does not need to be exported
  • RAPIDS_VERSION_NUMBER = needs to be exported, for rapids-upload-docs (code link in gha-tools)

For all the other scripts that are not docs builds, this PR only introduces NEW variables, so I think not exporting them is slightly safer.

Just pushed 16367a5 doing that. If that works, I'll make similar changes in the other PRs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This went fine in CI... docs built successfully and I see the expected versions of things in the environment. Docs look ok at the preview too: https://downloads.rapids.ai/ci/cudf/pull-request/17013/16367a5/docs/libcudf/html/index.html

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these claims I made about the variables not needing to be exported were incorrect. Addressed in #17042 (and updated across all the other PRs for this issue).


rapids-logger "Generate C++ testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"
Expand All @@ -31,7 +33,10 @@ rapids-print-env

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
libcudf libcudf_kafka libcudf-tests libcudf-example
"libcudf=${RAPIDS_VERSION_MAJOR_MINOR}" \
"libcudf_kafka=${RAPIDS_VERSION_MAJOR_MINOR}" \
"libcudf-tests=${RAPIDS_VERSION_MAJOR_MINOR}" \
"libcudf-example=${RAPIDS_VERSION_MAJOR_MINOR}"

rapids-logger "Check GPU usage"
nvidia-smi
4 changes: 3 additions & 1 deletion ci/test_java.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"

rapids-logger "Generate Java testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"
Expand All @@ -30,7 +32,7 @@ CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
libcudf
"libcudf=${RAPIDS_VERSION_MAJOR_MINOR}"

rapids-logger "Check GPU usage"
nvidia-smi
Expand Down
5 changes: 4 additions & 1 deletion ci/test_notebooks.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"

rapids-logger "Generate notebook testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"
Expand All @@ -30,7 +32,8 @@ PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)
rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
cudf libcudf
"cudf=${RAPIDS_VERSION_MAJOR_MINOR}" \
"libcudf=${RAPIDS_VERSION_MAJOR_MINOR}"

NBTEST="$(realpath "$(dirname "$0")/utils/nbtest.sh")"
pushd notebooks
Expand Down
5 changes: 4 additions & 1 deletion ci/test_python_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"

rapids-logger "Generate Python testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"
Expand Down Expand Up @@ -38,4 +40,5 @@ rapids-print-env
rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
cudf libcudf
"cudf=${RAPIDS_VERSION_MAJOR_MINOR}" \
"libcudf=${RAPIDS_VERSION_MAJOR_MINOR}"
6 changes: 5 additions & 1 deletion ci/test_python_other.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,14 @@ cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../
# Common setup steps shared by Python test jobs
source ./ci/test_python_common.sh test_python_other

RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
dask-cudf cudf_kafka custreamz
"dask-cudf=${RAPIDS_VERSION_MAJOR_MINOR}" \
"cudf_kafka=${RAPIDS_VERSION_MAJOR_MINOR}" \
"custreamz=${RAPIDS_VERSION_MAJOR_MINOR}"

rapids-logger "Check GPU usage"
nvidia-smi
Expand Down
Loading