Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: merge_parquet #97

Merged
merged 6 commits into from
Jul 19, 2024
Merged

feat: merge_parquet #97

merged 6 commits into from
Jul 19, 2024

Conversation

zbilodea
Copy link
Collaborator

No description provided.

@zbilodea zbilodea linked an issue Jun 17, 2024 that may be closed by this pull request
@zbilodea zbilodea changed the title Add merge_parquet feature feat: merge_parquet Jun 18, 2024
@ikrommyd
Copy link

ikrommyd commented Jul 19, 2024

Hi @zbilodea, should this be useable? I'd like to merge thousands of parquet files of order a few MBs each into one and I figured I'd be a good chance to try it out.

@zbilodea
Copy link
Collaborator Author

@ikrommyd
Tied up a few loose ends, should be worth testing now! But FYI it pages through files and doesn't write in batches yet, sorry!

@zbilodea zbilodea merged commit acdee53 into main Jul 19, 2024
13 checks passed
@zbilodea zbilodea deleted the feat-add-merge-parquet branch July 19, 2024 16:36
@zbilodea zbilodea restored the feat-add-merge-parquet branch July 22, 2024 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feature request: parquet merging
2 participants