feat: Optionally specify a max cost that will halt exchange once breached #225
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
michaelneale
approved these changes
Nov 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When creating autonomous use cases I often know a somewhat statistical upper bound where if a certain number of tokens/cost have been used then it will probably never solve the problem given to it. Most likely it is looping in some way.
To avoid blowing out costs this PR allows us to specify a
--max-costoption on the cli that once the exchange has received an output from the LLM and the cost is exceeded then no further information will be sent to the LLM, the session will pause and the user will see a message.By allowing us to short-circuit wayward sessions it opens up the option to have 'spend budgets' where we are willing to pay a certain amount to try to solve a problem. Then if the first approach fails we can try alternative approaches/prompts in a programatic way all within a budget which can increase the chance of success and protect ROI.