Openai compatible gauntlet #1017
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lm-foundry into migrate_subclasses_to_foundry
…lm-foundry into migrate_subclasses_to_foundry
…lm-foundry into migrate_subclasses_to_foundry
…lm-foundry into migrate_subclasses_to_foundry
…foundry into openai_compatible_gauntlet
…foundry into openai_compatible_gauntlet
maxisawesome
requested changes
Apr 19, 2024
| early_stopping_criteria: | ||
| - "\n\n" | ||
| - "Question:" | ||
| # - label: mmlu |
There was a problem hiding this comment.
Should these be uncommented or deleted?
| cd llm-foundry/scripts | ||
| composer eval/eval.py /mnt/config/parameters.yaml | ||
|
|
||
| # Mosaic Cloud will use run_name (with a unique suffix) to populate the env var $RUN_NAME | ||
| run_name: mpt-eval | ||
| name: mpt-eval-logging |
There was a problem hiding this comment.
Is this a temporary change or permanent?
There was a problem hiding this comment.
goes for whole file I think?
| name: api-eval | ||
| cluster: r1z1 # replace with your cluster here! | ||
| gpu_num: 8 # | ||
| gpu_type: a100_80gb # |
| if fsdp_config and model_cfg.model.get('load_in_8bit', False): | ||
| raise ValueError( | ||
| 'The FSDP config block is not supported when loading ' + | ||
| 'Hugging Face models in 8bit.') |
There was a problem hiding this comment.
This doesn't seem openai gauntlet related but prob should be merged anyway?
| name: hotpotqa | ||
| context_length: 65536 | ||
| section: beginning | ||
| split: test |
There was a problem hiding this comment.
Probably don't want to delete these do we?
| model_name: gpt-3.5-turbo | ||
| - | ||
| model_name: openai/davinci | ||
| model_name: openai/gpt-3.5-turbo-instruct |
There was a problem hiding this comment.
del old gpt-3.5-turbo throughout the codebase for instruct?
| @@ -114,8 +114,7 @@ | |||
| ] | |||
|
|
|||
| extra_deps['openai'] = [ | |||
| 'openai==1.3.8', | |||
| 'tiktoken==0.4.0', | |||
| 'openai==1.3.8', 'tiktoken==0.4.0', 'google-generativeai' | |||
There was a problem hiding this comment.
Should google-generativeai be here or it's own extra_deps category?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
OpenAI run:
api-eval-Ik2iMA