The spaces here are probably not necessary for consistency since this gets hashed.

flake8 fails when I try to commit because of these spaces, that's why I included them. What should I do instead?

Is this the best way for getting the unique rows? It seems very complex and expensive.

What if we use their built-in .unique()? (Please test if it flattens to the data or not) https://huggingface.co/docs/datasets/v3.1.0/en/package_reference/main_classes#datasets.Dataset.unique

Another concern is memory usage from having to load the entire data in memory like in your method.

From what I saw online, there is no inbuilt method in huggingface datasets to get the unique rows of a dataset (see this. Their inbuilt .unique() function takes in a column and provides a sorted list of unique values for that row.

Regarding whether there is an easier way, if there is no inbuilt function I think this might be the easiest way.

Regarding the cost of memory, this method only gets activated in areas of the code where the data is already fully loaded (for pretraining_datasets it is not activated). Since we are creating a list of unique rows, we are using at most roughly 2x the dataset size as memory (this includes the space taken by the original dataset).

Regarding how expensive/fast the method is, one thing I could do to make it faster is to directly use a hashing algorithm rather than converting every row to a JSON. Originally I was converting a row to a JSON because it makes it hashable, but I could simply use a hashing algorithm which should remove the overhead of converting to a JSON and back.

Please let me know what you think so I can start implementing those changes.

Their inbuilt .unique() function takes in a column and provides a sorted list of unique values for that row.

Hm I see. I misunderstood the docs and thought if given multiple columns, it would adjust.

Regarding how expensive/fast the method is, one thing I could do to make it faster is to directly use a hashing algorithm rather than converting every row to a JSON. Originally I was converting a row to a JSON because it makes it hashable, but I could simply use a hashing algorithm which should remove the overhead of converting to a JSON and back.

By expensive, I mean more on computation cost as it gets converted between multiple data types.

How about using pandas with their df.drop_duplicates? You should be able to convert between Dataset and DataFrame.

Yeah that can also work. I'll try it out and commit the changes to the PR soon.

I tried using df.drop_duplicates. Initially it fails when the cell content is a list, because it is not hashable. From my research the most reasonable way to proceed would be to cast the whole dataframe as string and then proceed. Something like this:

    unique_indices = df.astype(str).drop_duplicates().index
    deduplicated_df = df.loc[unique_indices]
    deduplicated_dataset = Dataset.from_pandas(deduplicated_df, preserve_index=False)

But this would be fairly expensive because apparently df.astype(str) will create a duplicate of the dataset. Instead, we can use hashing, which would be more memory efficient for our task, especially for large datasets. Here we would only need to store one hash per row, and one integer:

seen_hashes = set()
unique_indices = []

for idx, row in enumerate(dataset):
    row_hash = hash(str(row))
    if row_hash not in seen_hashes:
        seen_hashes.add(row_hash)
        unique_indices.append(idx)

deduplicated_dataset = dataset.select(unique_indices)

I changed it to the method where I hash, using sha256 to have enough expressiveness for diverse dataset sizes. Using this method also provides us with the benefit of being able to remove duplicates between train_dataset and eval_dataset because we can use the same set for both. This is what I did in the last commit (7-8).

Note that this file avoids pylint issues between other files, as there were repeating constants between some tests. @winglian