Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix request settings and stop words in HF provider #14504

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

JonasHelming
Copy link
Contributor

fixed #14503

What it does

  • Huggingface provider respects request settings
  • Added Huggingface default settings (the stop words are very common)
  • Remove all configured stop words from response

How to test

  • Set up a HuggingFace Model
  • e.g.: "Qwen/Qwen2.5-Coder-32B-Instruct"
  • Configure it for the universal chat agent
  • Ask it a question
  • Observe that always end with '<|im_end|>'
  • Change HF defaults in the model:
   protected getDefaultSettings(): Record<string, any> {
        return {
            max_new_tokens: 2024,
            stop: ['<|endoftext|>', '<eos>', '<|im_end|>']
        };
    }
  • Observe that it does not end with '<|im_end|>' anymore
  • Turn streaming of in the HuggingFaceModel
  • Same test
  • Turn streaming on in the HuggingFaceModel
  • Remove default override
  • Add settings to the universal-chat-agent
   protected override getLlmSettings(): { [key: string]: unknown } | undefined {
      return {
         max_new_tokens: 2024,
         stop: ['<|im_end|>'],
      };
   }
   
  • Observe that it does not end with '<|im_end|>' anymore
  • and without streaming

Follow-ups

  • We must think about how we want to allow the user configure these settings (per model, per provider, per agent, per prompt?)
  • If we start to really use the settings, we need a compatibility check for all provider APIs. Most parameters are valid for all of them, but not all of them

Review checklist

Reminder for reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Waiting on reviewers
Development

Successfully merging this pull request may close these issues.

Request Settings in Huggingface Provider are broken
1 participant