Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add allowParallel option #63

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Conversation

nh2
Copy link

@nh2 nh2 commented Jun 16, 2024

Draft PR for in-review libE57Format feature to allow parallel read access to the same E57 handle, e.g. with Python's ThreadPoolExecutor: asmaloney/libE57Format#292

TODO

@dancergraham
Copy link
Collaborator

Looks interesting! It would be cool to have a unit test for this and as noted it is important to remove the submodule change before merging. What are the performance implications?

@dancergraham
Copy link
Collaborator

There may also be some work required to bring pye57 up to date with other changes in Libe57format, particularly if this lands as part of a major revision.

@nh2
Copy link
Author

nh2 commented Jun 17, 2024

What are the performance implications?

The only one we know of is that you can now use multiple threads for reading data out of the same E57.

Which can produce big speedups when being IO-bound by a single thread (e.g. when on a striped RAID or network file system, reading with N threads can be up to N times faster).

(Of course one could also open the same E57 multiple times to get parallel read access, but it's less convenient because there are more open file handles, and one needs to write the code in a coarser way instead of being able to use it locally in a function which has an already-opened E57.)

@dancergraham
Copy link
Collaborator

free threading is available as a build flag on python 3.13 - I wonder how that would interact with this feature?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants