Improve multi-CTA algorithm #492
Conversation
tfeher
added
improvement
|
/ok to test |
|
@anaruse there are some unsigned commits, that blocks CI from testing the changes automatically. To fix this issue, could you rebase the PR? |
|
/ok to test |
…when the number of results is large Fix some issues Fix lower recall issue with new multi-cta algo Removing redundant code and changing some parameters Update cpp/src/neighbors/detail/cagra/search_plan.cuh Co-authored-by: Tamas Bela Feher <[email protected]> Remove an unnecessary line and satisfy clang-format
anaruse
force-pushed
the
improved_multi_cta_algo
branch
from
3dce160 to
6223fd2
Compare
|
/merge |
|
/ok to test |
|
/ok to test |
|
/merge |
|
/ok to test |
|
/merge |
|
We are on the brink of missing code freeze for this PR. Please anyone reading this, don't click the "update" button. It inserts a merge commit, which reruns CI in its entirety and this is not needed to merge the PR. We can re-run individual flaky tests that fail without having to rerun the entire CI (the former takes minutes and the latter can take several hours). |
|
@anaruse @tfeher CI seems to be running successfully for other PRs but the gtests seem to be consistently timing out for this PR. As far as I can tell, there's no updates to any of the tests, in this PR, but the timeouts don't seem flaky, they seem isolated to these changes, somehow. We are pushing back code freeze by 1 day. Do you guys think we can still make this in time for 24.12? |
Handle the case when the search result contains invalid indices when building the updated graph in add_nodes. For debugging purposes, fail if any invalid indices found; in future, we can replace RAFT_FAIL with RAFT_LOG_WARN to make the add_nodes routine more robust.
|
I took the liberty to add a workaround to add_nodes, which handles the case when CAGRA search doesn't return enough valid indices. With this, the tests should fail with a descriptive message in place of the segfault. |
|
/ok to test |
|
The cause has not yet been identified, but it seems that this problem only occurs when the dimensionality of the dataset is 1. |
It has been reported that when the number of search results is large, for example 100, using the multi-CTA algorithm can cause a decrease in recall. This PR is intended to alleviate this low recall issue.
close #208