Ability to limit LLM call count when tools are used #1004
se-roberthanson
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I created a tool that allows the LLM to trigger a "place search". And if I give it specific instructions I can cause Sprint AI to make many LLM calls.
Example prompt:
"think step by step.
you can perform sequential operations.
search for places names after each color of the rainbow, performing only one search at a time.
if a search returns zero results, then skip the next color.
if a search returns at least 1 result, search places for the next color."
In my test case Spring AI made 7 calls to OpenAI. The first 6 calls resulted in a tool execution (search for "red", "orange", etc.) and the 7th call provided the combined response.
My concern is that a knowledgeable user can force a loop that will results in a very high OpenAI bill for me.
Ideally there would be some interceptor so that I can make the decision for each iteration of the tool loop (as well as ready the token counts), but even something as simple as a property that allows me to limit the number of calls made to the LLM would work.
Let me know if this already exists and I missed it.
Beta Was this translation helpful? Give feedback.
All reactions