New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Define SF100k data set #307

Open

szarnyasg opened this issue Jun 19, 2021 · 1 comment

Labels

enhancement performance

Member

szarnyasg commented Jun 19, 2021

Our current largest data set is SF30k. We should introduce SF100k and set the number of persons to achieve the size of 100TiB (in CSVs).

szarnyasg added enhancement performance labels

szarnyasg mentioned this issue

Introduce SF100k data set #363

Closed

szarnyasg added this to the Milestone 5 milestone

szarnyasg removed this from the Milestone 5 milestone

Member Author

szarnyasg commented Oct 14, 2022

The trick to count the size of the data set is to use the s4cmd command-line tool:

$ s4cmd du -r \
    s3://${BUCKET_NAME}/results/${JOB_NAME}/runs/${RUN_TIMESTAMP}/social_network/graphs/csv/bi/composite-merged-fk/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment