Indeed we do need .cpu().detach() in encode_candidate():

[kaisugi]

This is not a bug report but a personal reply to the comment "# TODO: why do we need cpu here?"

BLINK/blink/biencoder/biencoder.py

Lines 135 to 144 in 5fe254d

	def encode_candidate(self, cands):
	token_idx_cands, segment_idx_cands, mask_cands = to_bert_input(
	cands, self.NULL_IDX
	)
	_, embedding_cands = self.model(
	None, None, None, token_idx_cands, segment_idx_cands, mask_cands
	)
	return embedding_cands.cpu().detach()
	# TODO: why do we need cpu here?
	# return embedding_cands

This func is repeatedly used for each batch when we run eval_biencoder.py, and if we stay the batch embeddings in GPU, we have to consume GPU memory every time, causing CUDA out of memory error. Therefore .cpu().detach() is required to free GPU memory. Note that the entire candidates are on GPU in get_topk_predictions() later, so we don't need to load GPU during this time.

If some of you have free time, please delete this misleading comment.

[abhinavkulkarni]

@HelloRusk: Not sure if this has anything to do with CUDA versions, but I found that the GPU memory is not cleared unless embedding_cands.cpu() is reassigned to embedding_cands.

This is, in my case I wasn't able to clear GPU memory by return embedding_cands.cpu().detach(), but rather by modifying it to the following:

embeddings =  embedding_cands.cpu().detach() 
del embedding_cands
torch.cuda.empty_cache()
return embeddings

Perhaps you don't have to explicitly garbage college by calling torch.cuda.empty_cache() explicitly, but you do need to del the GPU tensor in order to free up the memory on GPU. Simply calling .cpu() on it will create a CPU copy but not vacate GPU memory.