Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Diskann Benchmarking Wrapper #260

Open
wants to merge 103 commits into
base: branch-25.02
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
8e8d3c1
initial commit
tarang-jain Jul 29, 2024
e937ebd
merge 24.08
tarang-jain Jul 29, 2024
0bbbf0d
make build
tarang-jain Jul 29, 2024
02084e2
Merge branch 'branch-24.08' of https://github.com/rapidsai/cuvs into …
tarang-jain Jul 29, 2024
64f1d60
Merge branch 'branch-24.10' into diskann-wrapper
tarang-jain Jul 29, 2024
706f22e
update wrapper
tarang-jain Jul 31, 2024
3ea499b
Merge branch 'branch-24.08' of https://github.com/rapidsai/cuvs into …
tarang-jain Jul 31, 2024
e0aab8f
diskann_memory working
tarang-jain Aug 1, 2024
17c5510
Merge branch 'branch-24.08' of https://github.com/rapidsai/cuvs into …
tarang-jain Aug 1, 2024
f426df9
Merge branch 'branch-24.10' of https://github.com/rapidsai/cuvs into …
tarang-jain Aug 1, 2024
a7bdd33
make compile
tarang-jain Aug 3, 2024
d2442ca
Merge branch 'diskann-wrapper' of https://github.com/tarang-jain/cuvs…
tarang-jain Aug 3, 2024
dbc84cc
rm num_threads_
tarang-jain Aug 5, 2024
7e37218
FEA Add cuvs-bench to dependencies and conda environments
dantegd Aug 5, 2024
b2aef6d
FIX add missing deps
dantegd Aug 5, 2024
b9762d5
Merge branch 'fea-add-bench-deps' of https://github.com/dantegd/cuvs …
tarang-jain Aug 5, 2024
bf75242
FIX version and other improvements
dantegd Aug 6, 2024
a8bcdef
FEA Add cuvs_bench.run
dantegd Aug 6, 2024
3818da9
update patch;build command
tarang-jain Aug 6, 2024
cd8bfe5
Merge branch 'cuvsbench-run' of https://github.com/dantegd/cuvs into …
tarang-jain Aug 6, 2024
ec6d70c
FIX some cuvs_bench python build dependencies
dantegd Aug 6, 2024
c9f797a
Merge branch 'cuvsbench-run' of https://github.com/dantegd/cuvs into …
tarang-jain Aug 6, 2024
585ad53
FIX add missing algorithms.yaml
dantegd Aug 6, 2024
441ab2a
Merge branch 'cuvsbench-run' of https://github.com/dantegd/cuvs into …
tarang-jain Aug 6, 2024
33b075d
working mem index
tarang-jain Aug 7, 2024
81c92e6
Merge branch 'branch-24.10' of https://github.com/rapidsai/cuvs into …
tarang-jain Aug 7, 2024
11545c3
Merge branch 'rapidsai:branch-24.10' into diskann-wrapper
tarang-jain Aug 7, 2024
ffea663
remove base_set warning
tarang-jain Aug 8, 2024
9c1cddc
Merge branch 'branch-24.10' of https://github.com/rapidsai/cuvs into …
tarang-jain Aug 8, 2024
96d5642
Merge branch 'diskann-wrapper' of https://github.com/tarang-jain/cuvs…
tarang-jain Aug 8, 2024
a7eb787
pull upstream
tarang-jain Sep 19, 2024
4cbe7b1
merge 24.10
tarang-jain Oct 1, 2024
63621f4
revert some changes
tarang-jain Oct 1, 2024
a890ac5
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 1, 2024
450dcee
revert
tarang-jain Oct 1, 2024
b0f4b57
revert
tarang-jain Oct 1, 2024
8cd6c40
revert
tarang-jain Oct 1, 2024
d658856
update dependencies
tarang-jain Oct 2, 2024
d1e4101
Merge branch 'branch-24.10' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 2, 2024
2e080c6
update diff
tarang-jain Oct 3, 2024
8c6a178
Merge branch 'branch-24.10' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 3, 2024
626dc17
Merge branch 'branch-24.12' into diskann-wrapper
tarang-jain Oct 3, 2024
3d15882
style
tarang-jain Oct 3, 2024
a72165c
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 3, 2024
9c202f2
Merge branch 'diskann-wrapper' of https://github.com/tarang-jain/cuvs…
tarang-jain Oct 3, 2024
58a729c
diskann_memory working
tarang-jain Oct 8, 2024
2412b70
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 8, 2024
3a56402
ssd wrapper working, cuvs_vamana DOES NOT BUILD
tarang-jain Oct 15, 2024
0b39d5b
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 15, 2024
92ec474
builds now
tarang-jain Oct 15, 2024
333539f
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 15, 2024
a13bf1a
rm bug
tarang-jain Oct 15, 2024
df54939
add beam_width, rm dbg statements
tarang-jain Oct 15, 2024
93b2620
style
tarang-jain Oct 15, 2024
54385ab
updates after PR reviews, replace thread_pool with omp pragma
tarang-jain Oct 16, 2024
c35d899
codespell
tarang-jain Oct 16, 2024
31d846a
sync stream in vamana serialize
tarang-jain Oct 16, 2024
61e00c7
re-enable warnings
tarang-jain Oct 17, 2024
0bf43e9
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 17, 2024
0a6b094
deps in conda recipe
tarang-jain Oct 17, 2024
645d84b
host deps
tarang-jain Oct 17, 2024
d6897cc
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 17, 2024
396a589
Update conda/recipes/cuvs_bench/meta.yaml
tarang-jain Oct 17, 2024
d325698
Update conda/recipes/cuvs_bench_cpu/meta.yaml
tarang-jain Oct 17, 2024
03a1e09
arch dependendent diskann deps
tarang-jain Oct 18, 2024
f061f27
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 18, 2024
f95aec7
arch specific deps
tarang-jain Oct 18, 2024
b667786
CMAKE_SYSTEM_PROCESSOR check
tarang-jain Oct 18, 2024
4aa513f
update cmake flags
tarang-jain Oct 19, 2024
17f723e
CMAKE_SYSTEM_PROCESSOR regex
tarang-jain Oct 19, 2024
6532914
diskann build params
tarang-jain Oct 19, 2024
1f168a8
dbg
tarang-jain Oct 19, 2024
2a5d1fb
rename script to cuvs_vamana.cu
tarang-jain Oct 21, 2024
173df8f
Merge branch 'branch-24.12' into diskann-wrapper
tarang-jain Oct 21, 2024
46e7728
rm dbg statement
tarang-jain Oct 21, 2024
1b03cf7
Merge branch 'diskann-wrapper' of https://github.com/tarang-jain/cuvs…
tarang-jain Oct 21, 2024
c131c52
do not link cuvs for diskann only targets
tarang-jain Oct 21, 2024
3d40d2d
link aio
tarang-jain Oct 24, 2024
c3a25fc
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Oct 24, 2024
6bebeb8
style
tarang-jain Oct 24, 2024
e254c9b
Merge branch 'branch-24.12' into diskann-wrapper
tarang-jain Oct 28, 2024
82e21e8
Merge branch 'branch-24.12' into diskann-wrapper
cjnolet Nov 4, 2024
de2bf84
Merge branch 'branch-24.12' of https://github.com/rapidsai/cuvs into …
tarang-jain Nov 13, 2024
6d3b32d
include utils.h
tarang-jain Nov 27, 2024
2506ef1
serialize dataset
tarang-jain Nov 28, 2024
16f35b3
debug
tarang-jain Nov 28, 2024
18f26a7
rm debug statements
tarang-jain Nov 29, 2024
663dfe0
style
tarang-jain Nov 29, 2024
4bb2b39
alloc strided dim on host
tarang-jain Nov 29, 2024
fd43711
size
tarang-jain Nov 29, 2024
30bcc6e
Merge branch 'diskann-wrapper' of https://github.com/tarang-jain/cuvs…
Dec 2, 2024
a10d834
include_dataset flag
Dec 2, 2024
4b396d7
Merge branch 'branch-24.12' into diskann-wrapper
tarang-jain Dec 2, 2024
c443944
Merge branch 'diskann-wrapper' of https://github.com/tarang-jain/cuvs…
Dec 2, 2024
ed8c9b6
Merge branch 'branch-24.12' into diskann-wrapper
tarang-jain Dec 3, 2024
f015264
rm cagra+diskann
Dec 4, 2024
6d6167d
Merge branch 'branch-24.12' into diskann-wrapper
tarang-jain Dec 4, 2024
9c2185b
Merge branch 'branch-24.12' into diskann-wrapper
tarang-jain Dec 5, 2024
2b758d3
Merge branch 'diskann-wrapper' of https://github.com/tarang-jain/cuvs…
tarang-jain Dec 10, 2024
b7ba35b
update copyright
tarang-jain Dec 10, 2024
63e02ff
Merge branch 'branch-25.02' into diskann-wrapper
tarang-jain Dec 11, 2024
48a6a9d
Merge branch 'branch-25.02' into diskann-wrapper
cjnolet Dec 12, 2024
fd429d2
style
tarang-jain Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
FEA Add cuvs-bench to dependencies and conda environments
  • Loading branch information
dantegd committed Aug 5, 2024
commit 7e3721809c8142ae5daa31e59fb0a1ae74b7a04e
43 changes: 43 additions & 0 deletions conda/environments/bench_ann_cuda-118_arch-aarch64.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
- benchmark>=1.8.2
- c-compiler
- clang-tools=16.0.6
- clang==16.0.6
- cmake>=3.26.4,!=3.30.0
- cuda-nvtx=11.8
- cuda-profiler-api=11.8.86
- cuda-version=11.8
- cudatoolkit
- cxx-compiler
- cython>=3.0.0
- gcc_linux-aarch64=11.*
- glog>=0.6.0
- h5py>=3.8.0
- hnswlib=0.7.0
- libcublas-dev=11.11.3.6
- libcublas=11.11.3.6
- libcurand-dev=10.3.0.86
- libcurand=10.3.0.86
- libcusolver-dev=11.4.1.48
- libcusolver=11.4.1.48
- libcusparse-dev=11.7.5.86
- libcusparse=11.7.5.86
- matplotlib
- nccl>=2.9.9
- ninja
- nlohmann_json>=3.11.2
- nvcc_linux-aarch64=11.8
- openblas
- pandas
- pyyaml
- rmm==24.10.*,>=0.0.0a0
- sysroot_linux-aarch64==2.17
name: bench_ann_cuda-118_arch-aarch64
43 changes: 43 additions & 0 deletions conda/environments/bench_ann_cuda-118_arch-x86_64.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
- benchmark>=1.8.2
- c-compiler
- clang-tools=16.0.6
- clang==16.0.6
- cmake>=3.26.4,!=3.30.0
- cuda-nvtx=11.8
- cuda-profiler-api=11.8.86
- cuda-version=11.8
- cudatoolkit
- cxx-compiler
- cython>=3.0.0
- gcc_linux-64=11.*
- glog>=0.6.0
- h5py>=3.8.0
- hnswlib=0.7.0
- libcublas-dev=11.11.3.6
- libcublas=11.11.3.6
- libcurand-dev=10.3.0.86
- libcurand=10.3.0.86
- libcusolver-dev=11.4.1.48
- libcusolver=11.4.1.48
- libcusparse-dev=11.7.5.86
- libcusparse=11.7.5.86
- matplotlib
- nccl>=2.9.9
- ninja
- nlohmann_json>=3.11.2
- nvcc_linux-64=11.8
- openblas
- pandas
- pyyaml
- rmm==24.10.*,>=0.0.0a0
- sysroot_linux-64==2.17
name: bench_ann_cuda-118_arch-x86_64
39 changes: 39 additions & 0 deletions conda/environments/bench_ann_cuda-125_arch-aarch64.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
- benchmark>=1.8.2
- c-compiler
- clang-tools=16.0.6
- clang==16.0.6
- cmake>=3.26.4,!=3.30.0
- cuda-cudart-dev
- cuda-nvcc
- cuda-nvtx-dev
- cuda-profiler-api
- cuda-version=12.5
- cxx-compiler
- cython>=3.0.0
- gcc_linux-aarch64=11.*
- glog>=0.6.0
- h5py>=3.8.0
- hnswlib=0.7.0
- libcublas-dev
- libcurand-dev
- libcusolver-dev
- libcusparse-dev
- matplotlib
- nccl>=2.9.9
- ninja
- nlohmann_json>=3.11.2
- openblas
- pandas
- pyyaml
- rmm==24.10.*,>=0.0.0a0
- sysroot_linux-aarch64==2.17
name: bench_ann_cuda-125_arch-aarch64
39 changes: 39 additions & 0 deletions conda/environments/bench_ann_cuda-125_arch-x86_64.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
- benchmark>=1.8.2
- c-compiler
- clang-tools=16.0.6
- clang==16.0.6
- cmake>=3.26.4,!=3.30.0
- cuda-cudart-dev
- cuda-nvcc
- cuda-nvtx-dev
- cuda-profiler-api
- cuda-version=12.5
- cxx-compiler
- cython>=3.0.0
- gcc_linux-64=11.*
- glog>=0.6.0
- h5py>=3.8.0
- hnswlib=0.7.0
- libcublas-dev
- libcurand-dev
- libcusolver-dev
- libcusparse-dev
- matplotlib
- nccl>=2.9.9
- ninja
- nlohmann_json>=3.11.2
- openblas
- pandas
- pyyaml
- rmm==24.10.*,>=0.0.0a0
- sysroot_linux-64==2.17
name: bench_ann_cuda-125_arch-x86_64
45 changes: 45 additions & 0 deletions dependencies.yaml
Original file line number Diff line number Diff line change
@@ -21,6 +21,18 @@ files:
- test_py_cuvs
- cupy
- rust
bench_ann:
output: conda
matrix:
cuda: ["11.8", "12.5"]
arch: [x86_64, aarch64]
includes:
- rapids_build
- cuda
- cuda_version
- develop
- bench
- bench_python
test_cpp:
output: none
includes:
@@ -89,6 +101,20 @@ files:
- test_python_common
- test_py_cuvs
- cupy
py_build_cuvs_bench:
output: pyproject
pyproject_dir: python/cuvs_bench
extras:
table: build-system
includes:
- rapids_build
py_run_cuvs_bench:
output: pyproject
pyproject_dir: python/cuvs_bench
extras:
table: project
includes:
- bench_python
channels:
- rapidsai
- rapidsai-nightly
@@ -429,3 +455,22 @@ dependencies:
- output_types: [conda, requirements, pyproject]
packages:
- scikit-learn
bench:
common:
- output_types: [conda, pyproject, requirements]
packages:
- hnswlib=0.7.0
- nlohmann_json>=3.11.2
- glog>=0.6.0
- h5py>=3.8.0
- benchmark>=1.8.2
- openblas
- *rmm_unsuffixed
bench_python:
common:
- output_types: [conda]
packages:
- matplotlib
- pandas
- pyyaml
- pandas
Loading