Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PARQUET-2477: close readers opened by ParquetFileWriter #1351

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

packyan
Copy link

@packyan packyan commented May 18, 2024

Files opened by appendFile method of ParquetFileWriter should be closed correctly.

Make sure you have checked all steps below.

Jira

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines
    from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Style

  • My contribution adheres to the code style guidelines and Spotless passes.
    • To apply the necessary changes, run mvn spotless:apply -Pvector-plugins

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain Javadoc that explain what it does

@packyan packyan changed the title PARQUET-2477: close readers opened by ParquetFileWriter. PARQUET-2477: close readers opened by ParquetFileWriter May 18, 2024
@packyan packyan force-pushed the bugfix_unclosed_reader_leak branch from 7e54077 to a587b49 Compare May 18, 2024 13:14
Currently, files opened by ParquetFileWriter via appendFile are not closed.
This change addresses this by explicitly closing the readers.
@packyan packyan force-pushed the bugfix_unclosed_reader_leak branch from a587b49 to b5c09b9 Compare May 18, 2024 13:15
@@ -1433,12 +1436,14 @@ public void endBlock() throws IOException {
public void appendFile(Configuration conf, Path file) throws IOException {
try (ParquetFileReader reader = ParquetFileReader.open(conf, file)) {
reader.appendTo(this);
readers.add(reader);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, ParquetFileReader implements AutoCloseable which should be closed automatically when the current scope has ended. Could you help investigate why it is not called on Windows? I have concern that this fix might hide the root cause forever.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants