Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: this refactors execution to properly use the process pool #352

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

fix: this refactors execution to properly use the process pool #352

wants to merge 8 commits into from
+236 −70

Commits on Mar 21, 2024

  1. fix: this refactors execution to properly use the proces pool 

    Prior to this change the process pool was created and destroyed
for every configuration, this would cause the cpu/memory to
thrash and improperly allocate task to avaialble cpu, resulting
in sometimes 25% utilization of available cpu resources

The change corrects this, as well as tackles memory problems,
by writing temporary results to disk and then reading them back
at the end of the simulation.

This is non configurable in this commit, and can also result in
loading too much memory, as it does not include the ability to
progressively or lazy load data into the final dataframe to
complete the simulation.

This commit is a WIP fixes cadCAD-org#351
    @zcstarr
    zcstarr committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    cee080b View commit details
    Browse the repository at this point in the history

  2. fix: use dill instead of pickle to support lambda

    @zcstarr
    zcstarr committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    4f2a539 View commit details
    Browse the repository at this point in the history

  3. fix: rm dead code

    @zcstarr
    zcstarr committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    30b0729 View commit details
    Browse the repository at this point in the history

  4. feat: add temporary usage sketch

    @zcstarr
    zcstarr committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    0efa2e2 View commit details
    Browse the repository at this point in the history

  5. feat: add support for lazy flattening of data structures

    @zcstarr
    zcstarr committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    996dd5c View commit details
    Browse the repository at this point in the history

  6. feat: add support for lazy evaluation during simulation processing 

    This feature refactors the parallel processing to use lazy evaluation
When simulation data grows, keeping the data in memory causes
memory bloat, which causes too Ram to be consumed.

The refactor uses lazy_flattens to keep the memory Python consumes
low, the change was able to reduce consumption by a factor of 10
in many cases.

The change also writes temporary files to disk before lazily
rearranging the simulation results to fit the cadCAD expected format
for a regular pandas dataframe.
    @zcstarr
    zcstarr committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    024e91c View commit details
    Browse the repository at this point in the history

  7. chore: add temporary memory_profiler

    @zcstarr
    zcstarr committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    d9dce51 View commit details
    Browse the repository at this point in the history

  8. fix: check for additional obj existence

    @zcstarr
    zcstarr committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    9f42e1f View commit details
    Browse the repository at this point in the history