BUG rerank single document with bge-reranker-v2-gemma trigger exception

[stdweird]

Describe the bug

when reranking a single document with bge-reranker-v2-gemma, you get

  File "/home/xinference/.local/lib/python3.11/site-packages/xinference/model/rerank/core.py", line 219, in rerank
    docs = [
  File "/home/xinference/.local/lib/python3.11/site-packages/xinference/model/rerank/core.py", line 222, in <listcomp>
    relevance_score=float(similarity_scores[arg]),
    ^^^^^^^^^^^^^^^^^
TypeError: [address=0.0.0.0:40107, pid=862388] 'float' object is not subscriptable

(line numbers might be a bit off, since i added some debug lines).

The reason seems to be that a single document gives a similarity_scores that is not an array, but a float.

I previously thought that the float in the exception was about arg (found that via some stackoverflow post), but it's not the issue. It's the following example:

>>> a=0.5
>>> a[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'float' object is not subscriptable

[codingl2k1]

The bge-reranker-v2-gemma model is a LLM-based model. We use the FlagLLMReranker from FlagEmbedding package to inference. From the code, it shows the compute_score should return a list of float, but in your case it returns a float. I think there may have some bugs in the FlagEmbedding.

[stdweird]

@codingl2k1 it's right there in their examples: https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/llm_reranker#for-normal-reranker-bge-reranker-base--bge-reranker-large--bge-reranker-v2-m3-

from FlagEmbedding import FlagReranker
reranker = FlagReranker('BAAI/bge-reranker-v2-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation

score = reranker.compute_score(['query', 'passage'])
print(score) # -5.65234375

i'll open an issue for it

[stdweird]

see FlagOpen/FlagEmbedding#943

[codingl2k1]

Thanks. I will create a fix PR to fix this case in xinference.