drop_query_vec

[seanmacavaney]

fixes #23

WIP:

• np_retriever
• faiss_flat_retriever
• faiss_hnsw_retriever
• faiss_ivf_retriever
• scann_retriever
• torch_retriever
• voyager_retriever
• gar
• ladr

[seanmacavaney]

Verified that it copies the pointer not the value.

(Pdb) id(res.query_vec[0])
140657462180720
(Pdb) id(res.query_vec[1])
140657462180720
(Pdb) id(res.query_vec[999])
140657462180720
(Pdb) id(res.query_vec[1000])
140657462180816

[seanmacavaney]

R -> R ones, like gar and ladr, are tricker to pull all input columns through because we need to figure out which columns are the same across the query (e.g., 'query_vec') and which are specific to the retrieval result (e.g., 'text'). Something like:

qcols = {group[k].iloc[0] for k in group if group[k].nunique() == 1}

I wonder if this is something that DataFrameBuilder could support directly somehow. (Or perhaps the use case is too niche.)

[seanmacavaney]

@cmacdonald do the implementations so far meet your needs?

[cmacdonald]

LGTM

[cmacdonald]

R -> R ones, like gar and ladr, are tricker to pull all input columns through because we need to figure out which columns are the same across the query (e.g., 'query_vec') and which are specific to the retrieval result (e.g., 'text'). Something like:

qcols = {group[k].iloc[0] for k in group if group[k].nunique() == 1}

I wonder if this is something that DataFrameBuilder could support directly somehow. (Or perhaps the use case is too niche.)

The "adhoc rule" is that query columns (i.e. which should be common across a query) begin with a q.

There are some methods for detecting query columns (no these dont seem to implement the rule, per se):
https://github.com/terrier-org/pyterrier/blob/master/pyterrier/model.py#L50-L60

Other pieces of code have weird joins, e.g. https://github.com/terrier-org/pyterrier/blob/master/pyterrier/terrier/retriever.py#L440-L441 - at least some used pt.model.query_columnms: https://github.com/search?q=repo%3Aterrier-org%2Fpyterrier%20pt.model.query_columns&type=code

[cmacdonald]

I'd be tempted to merge this for now.

[seanmacavaney]

I'd rather be able to sort it completely instead of leaving it half-done. Implementations for the remaining ones are in progress.