Skip to content
This repository has been archived by the owner on Sep 24, 2024. It is now read-only.

Commit

Permalink
Clarification on dataset format
Browse files Browse the repository at this point in the history
  • Loading branch information
aittalam committed Apr 23, 2024
1 parent c5e96ec commit 11355e3
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/source/evaluation_concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ In the `evaluation` directory, there are sample files for running evaluation on

Evaluation relies on [Prometheus](https://github.com/kaistAI/Prometheus) as LLM judge. We internally serve it via [vLLM](https://github.com/vllm-project/vllm) but any other OpenAI API compatible service should work (e.g. llamafile via their `api_like_OAI.py` script).

Input datasets _must_ be in HuggingFace format. The code below shows how to convert Prometheus benchmark datasets and optionally save them as wandb artifacts:
Input datasets _must_ be saved as HuggingFace [datasets.Dataset](https://huggingface.co/docs/datasets/v2.19.0/en/package_reference/main_classes#datasets.Dataset). The code below shows how to convert Prometheus benchmark datasets and optionally save them as wandb artifacts:

```
import wandb
Expand Down

0 comments on commit 11355e3

Please sign in to comment.