BUG xinference部署bge rerank模型经常出现服务错误

[yaoyasong]

Describe the bug

用xinference0.10.2.post1版本部署了bge-reranker-v2-m3模型，但经常出现如下错误，并且经常会出现服务无法响应的问题，需要重启才能解决：

xinference.api.restful_api 22480 ERROR [address=0.0.0.0:44871, pid=22579] 'float' object is not subscriptable
Traceback (most recent call last):
File "/app/anaconda3/lib/python3.11/site-packages/xinference/api/restful_api.py", line 1067, in rerank
scores = await model.rerank(
^^^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
...
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/model.py", line 79, in wrapped_func
ret = await fn(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/model.py", line 418, in rerank
return await self._call_wrapper(
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/model.py", line 103, in _async_wrapper
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/model.py", line 333, in _call_wrapper
ret = await asyncio.to_thread(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/model/rerank/core.py", line 168, in rerank
docs = [
File "/app/anaconda3/lib/python3.11/site-packages/xinference/model/rerank/core.py", line 171, in
relevance_score=float(similarity_scores[arg]),
^^^^^^^^^^^^^^^^^

[qinxuye]

日志帮忙贴全，另外试下 0.10.3 还有没有问题。

[yaoyasong]

xinference.api.restful_api 22480 ERROR [address=0.0.0.0:44871, pid=22579] 'float' object is not subscriptable
Traceback (most recent call last):
File "/app/anaconda3/lib/python3.11/site-packages/xinference/api/restful_api.py", line 1067, in rerank
scores = await model.rerank(
^^^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/app/anaconda3/lib/python3.11/site-packages/xoscar/backends/pool.py", line 659, in send
result = await self._run_coro(message.message_id, coro)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
return await coro
File "/app/anaconda3/lib/python3.11/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 558, in on_receive
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
^^^^^^^^^^^^^^^^^
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/utils.py", line 45, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/model.py", line 79, in wrapped_func
ret = await fn(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/model.py", line 418, in rerank
return await self._call_wrapper(
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/model.py", line 103, in _async_wrapper
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/core/model.py", line 333, in _call_wrapper
ret = await asyncio.to_thread(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^
File "/app/anaconda3/lib/python3.11/site-packages/xinference/model/rerank/core.py", line 168, in rerank
docs = [
File "/app/anaconda3/lib/python3.11/site-packages/xinference/model/rerank/core.py", line 171, in
relevance_score=float(similarity_scores[arg]),
^^^^^^^^^^^^^^^^^

[codingl2k1]

这个错误是 FlagReranker 的问题，它方法签名写的是返回个 List[float]但实际上返回了个 float：https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/flag_reranker.py#L194

这个问题在xinference 新版已经部分切回 sentence transformers 了，你可以更新版本试试。

[yaoyasong]

0.10.3版本下没有问题

[stdweird]

@yaoyasong @codingl2k1 my apologies, but how was this fixed? i just ran into this with version 0.12

i see that on https://github.com/xorbitsai/inference/blob/main/xinference/model/rerank/core.py#L211 you convert the arg to an int, there must be a reason for that, and probably the same requirements is needed to use arg as an array index

so line below might have to be

relevance_score=float(similarity_scores[int(arg)]),

(and also in the else block)

[codingl2k1]

https://github.com/xorbitsai/inference/blob/main/xinference/model/rerank/core.py#L211

@aresnow1 Do you remember why you convert the arg to int? https://github.com/xorbitsai/inference/blob/main/xinference/model/rerank/core.py#L211

[codingl2k1]

@yaoyasong @codingl2k1 my apologies, but how was this fixed? i just ran into this with version 0.12

i see that on https://github.com/xorbitsai/inference/blob/main/xinference/model/rerank/core.py#L211 you convert the arg to an int, there must be a reason for that, and probably the same requirements is needed to use arg as an array index

so line below might have to be

relevance_score=float(similarity_scores[int(arg)]),

(and also in the else block)

Which model are you using? Could you provide the traceback?

[stdweird]

@codingl2k1 the bge gemma v2 one. but i do not always run into the issue

[stdweird]

@aresnow1 @codingl2k1 is it ok/safe to cast the arg to an int?

[stdweird]

it is a different cause, i opened #1775