Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question fixes #41

Merged
merged 1 commit into from
Oct 25, 2023
Merged

Question fixes #41

merged 1 commit into from
Oct 25, 2023

Conversation

wongjingping
Copy link
Collaborator

@wongjingping wongjingping commented Oct 25, 2023

Fixes to the following question/query pairs as per feedback:

  • count distinct pid instead of cid since question is asking for publications
  • clarify question on stopovers since it wasn't clear if don't want flights from/to JFK and stops=0 or if we don't want flights from/to JFK or stops=0. Updated the query accordingly as well.
  • clarify question on citing relationships

Tested with oa chat runner that it still works fine:

python main.py \
  -q data/questions_gen.csv \
  -o results/my_query_generator.csv \
  -g oa \
  -f prompts/prompt_openai.md \
  -m gpt-3.5-turbo-0613 \
  -p 5
...
Correct so far: 132/200 (66.00%): 100%|███████████████████████████████████████████████| 200/200 [00:41<00:00,  4.78it/s]
                exact_match   correct
query_category                       
date_functions     0.760000  0.760000
group_by           0.742857  0.742857
order_by           0.628571  0.714286
ratio              0.257143  0.342857
table_join         0.685714  0.714286
where              0.714286  0.714286
Average correct rate: 0.66

@rishsriv rishsriv merged commit 4978ae0 into main Oct 25, 2023
2 checks passed
@rishsriv rishsriv deleted the jp/qn_fix branch October 25, 2023 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants