Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rui/ds1000 #44

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open

rui/ds1000 #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
c2a2116
added length_calculator
rays1024 Apr 14, 2023
529edce
re-upload the implementation for llama, alpaca, santacoder
yilunzhao Apr 25, 2023
7caf0ea
Merge branch 'main' into yilunzhao/llm_implementation
yilunzhao Apr 25, 2023
9dbc8cd
try to pass CI check
yilunzhao Apr 26, 2023
06c7858
Merge branch 'main' into yilunzhao/llm_implementation
niansong1996 Apr 27, 2023
ce36f13
modify process_output function in exectutor.py to handle the case tha…
yilunzhao Apr 27, 2023
60fd89b
fix error related to llama eos_token
yilunzhao May 5, 2023
83546d9
add starcoder, gpt-neox-20b; test gpt-j-6b
yilunzhao May 6, 2023
86d3bd5
archieve code for NeurIPS exps; add gpt-4, pythia, replit, dolly, sta…
yilunzhao May 19, 2023
d10a59a
added dataset and executor for DS1000
Jun 28, 2023
a9df68b
Merge branch 'main' into rui/ds-1000
Jun 28, 2023
40c02b5
removed all .DS_Store
Jun 28, 2023
af3ffa2
added ds1000 prompt file
Jun 29, 2023
2bfee22
fixed missing file and added ds1000 prompt file
Jun 29, 2023
9e8b571
zipped ds1000_data and removed irrelevant files
Jun 29, 2023
fb089c0
class variable ds_data now initialize in init
rays1024 Jul 5, 2023
cfb10c7
Merge remote-tracking branch 'origin/yilunzhao/llm_implementation' in…
rays1024 Jul 5, 2023
adb4c79
fixed ds1000 gpt-neo-125M evaluation bugs
rays1024 Jul 5, 2023
39a4fd4
some code clean up
rays1024 Jul 5, 2023
0ecdb5f
included reference code in prompt
rays1024 Jul 5, 2023
5666a22
updated ds1000.yaml
rays1024 Jul 14, 2023
27e0e70
fixed evaluation error at 158/159th problem
rays1024 Jul 16, 2023
4ba669d
fixed evaluation bug
rays1024 Jul 16, 2023
dc4fc0c
fixed torch tensor bug
rays1024 Jul 21, 2023
62250c5
undo changes caused by wrong jsonargparse version
rays1024 Jul 25, 2023
f7e930b
fixed ds1000 saving issue
rays1024 Jul 27, 2023
1ed0607
Revert "fixed ds1000 saving issue"
rays1024 Jul 27, 2023
90cba4f
fixed result saving issue
rays1024 Jul 27, 2023
4229001
evaluating raw output without cutting off at keywords
rays1024 Jul 31, 2023
b787508
No custom instruction and using only DS1000 prompt
rays1024 Jul 31, 2023
73b9676
reverted raw output and uses keyword cutoff in execution
rays1024 Aug 1, 2023
d41eb3b
no cutoff at execution to check raw output
rays1024 Aug 1, 2023
5fad2f3
updated gitignore
rays1024 Aug 2, 2023
b3aa6ba
execution now preprocess model output
rays1024 Aug 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,3 +135,12 @@ debug-tmp/
wandb/
results/
.vscode/

# defined by Rui
.DS_Store
DS_1000/ds1000_data
results
data
NLP4Code_ds1000_outputs
NLP4Code_humaneval_outputs
raw_output_evaluation.py
6 changes: 6 additions & 0 deletions DS_1000/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Generate JSONL file and evaluation

- Cloned the repo from https://github.com/HKUNLP/DS-1000.git

- unzip `ds1000_data.zip` before preprocessing and evaluation
- run `preprocess.py` in DS1000 directory to generate JSONL files
Loading
Loading