Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing mmlu as valid_dataset even though readme encourages download #21

Open
jackfreeman88 opened this issue Dec 29, 2023 · 1 comment

Comments

@jackfreeman88
Copy link

(gpt5) [jfreeman@mir-81 promptbase]$ python -m promptbase mmlu
usage: main.py [-h] [--subject SUBJECT] [--list_subjects] {gsm8k,humaneval,math,drop,bigbench}
main.py: error: argument dataset: invalid choice: 'mmlu' (choose from 'gsm8k', 'humaneval', 'math', 'drop', 'bigbench')

Head of main.py :

import argparse
from promptbase.gsm8k import gsm8k
from promptbase.humaneval import humaneval
from promptbase.math import math
from promptbase.drop import drop
from promptbase.bigbench import bigbench
from promptbase.bigbench.consts import BIGBENCH_SUBJECTS

VALID_DATASETS = ["gsm8k", "humaneval", "math", "drop", "bigbench"]

:/

@riedgar-ms
Copy link
Collaborator

I'm looking in to this as part of #23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants