Nemotron fix #61
Nemotron fix #61
Conversation
for more information, see https://pre-commit.ci
|
|
||
| EVAL_MAP = { | ||
| 'base': { | ||
| 'default': 'nemotron/fewshot', |
There was a problem hiding this comment.
That's not really for the base model as it still uses the extra id tokens. Why do you need to add this file? If you only eval on math benchmarks, you don't need to specify the eval map stuff and our previous commands should work. If you want to eval on both math and other benchmarks, then you need to add other prompts
There was a problem hiding this comment.
This file was added because Nemotron-340B math evaluations required this eval_map file. Or we can do without it as well?
There was a problem hiding this comment.
It should work without it (there was a bug, but it should be fixed now). Let me know if you're running into problems
There was a problem hiding this comment.
If you do want to merge this, then please also add the prompts for code benchmarks + a proper base prompt (or just remove base). If not, let's just not merge this for now to not confuse people since we only want to have that file when multi-area eval is actually possible
There was a problem hiding this comment.
Let's not merge for now then since I haven't tested prompts with Nemotron models for different tasks.
Added eval map for Nemotron models.