Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Defer statepoint instantiation, unify reset_statepoint logic #497

Merged

Conversation

bdice
Copy link
Member

@bdice bdice commented Feb 6, 2021

Description

@vyasr this is an attempt to defer instantiation of the _StatePointDict class. Related to #484 (comment).

I also added an option to disable input data validation in SyncedCollection._update. I am moving the option to disable data validation into a separate PR.

Motivation and Context

Speeds up Project.open_job by deferring _StatePointDict.__init__ until Job.statepoint access.
This is a performance improvement for common access patterns (project iteration) for the same reasons as lazy loading, but benchmarks measuring "time to read statepoints" will see no change.

With deferred instantiation of the _StatePointDict, the profile of open_job looks identical for the current state of PR #484 and v1.6.0. As a consequence, we can conclude that the slowdown of open_job from v1.6.0 to the current state of PR #484 is entirely due to the hierarchical access of SyncedAttrDict.__setattr__.

Effect of the changes

With data validation disabled during _StatePointDict.load, the performance is closer to v1.6.0 but it won't match until we figure out how to improve SyncedAttrDict.__setattr__. Profiling a workspace of 1000 jobs, where I load the state point of every job:

The only other item causing slowdown relative to v1.6.0 is the call to Job._statepoint_filename, which appears to call os.path.join twice (once for the workspace, once for the state point filename) instead of once (directly computing the state point filename). Unless someone sees a way around that, I'm fine with ignoring that small impact.

@bdice bdice requested review from a team as code owners February 6, 2021 05:04
@bdice bdice requested review from b-butler and klywang and removed request for a team, b-butler and klywang February 6, 2021 05:04
@bdice bdice marked this pull request as draft February 6, 2021 05:04
@bdice bdice changed the base branch from master to feature/synced_collections February 6, 2021 05:04
@bdice bdice changed the title WIP: Synced Collections lazier statepoint init Experiment: Synced Collections lazier statepoint init Feb 6, 2021
@bdice bdice changed the title Experiment: Synced Collections lazier statepoint init Experiment: Synced Collections optimizations Feb 6, 2021
@bdice bdice self-assigned this Feb 6, 2021
@bdice bdice requested a review from vyasr February 6, 2021 05:27
@codecov
Copy link

codecov bot commented Feb 6, 2021

Codecov Report

Merging #497 (2911071) into feature/synced_collections (f0537c4) will increase coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@                      Coverage Diff                       @@
##           feature/synced_collections     #497      +/-   ##
==============================================================
+ Coverage                       78.11%   78.12%   +0.01%     
==============================================================
  Files                              63       63              
  Lines                            6973     6977       +4     
  Branches                         1302     1303       +1     
==============================================================
+ Hits                             5447     5451       +4     
  Misses                           1225     1225              
  Partials                          301      301              
Impacted Files Coverage Δ
signac/contrib/job.py 89.96% <100.00%> (+0.14%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f0537c4...2911071. Read the comment docs.

@vyasr
Copy link
Contributor

vyasr commented Feb 15, 2021

I think something along the lines of this change is fine. Adding a parameter to _update makes sense to me, but rather than implementing the parameter passing at the Job level, I think what would make sense is to add a parameter trust_source or similar to SyncedCollection that controls the argument to _update in _load. This parameter would indicate that when initializing a collection pointing to a pre-existing resource, we trust that the contents of that resource are well-formed. This will never be 100% true since, by construction, _load_from_resource will fail on a malformed underlying resource, but this parameter could control whether we need to validate data loaded from the resource.

The same change (whichever we decide on) should also be implemented for SyncedList.

Since this PR is meant to be an optimization, I should point out that some of the changes of _statepoint to statepoint will be less efficient with your changes. Any operation where a state point is modified before its data is loaded will be slower now because the load method will be called, whereas before it would not have been. This probably isn't typically performance-critical, on the other hand I don't see the minor consistency improvement of using the public property as a good enough reason to slow this down. This kind of optimization is precisely what typically justifies internal usage of an attribute rather than the public-facing property IMO.

@bdice
Copy link
Member Author

bdice commented Feb 15, 2021

I discussed more with @vyasr on a call. Takeaways:

  • Deferring the construction of _statepoint is fine, and it's okay to use only the public property statepoint in the Job.init method.
  • I need to refactor update_statepoint and the statepoint property setter to call reset_statepoint.
  • The reset_statepoint method needs to use the private _statepoint (and construct it, conditional on _statepoint_requires_init) to avoid triggering an unnecessary load.
  • In synced_collections, rename _update parameter to trust_source which will skip validation.
  • Refactor SyncedList to have a similar parameter trust_source.

Meanwhile, @vyasr will refactor the "attribute" accessors and make them only usable for a new child class JSONAttrDict.

@bdice bdice changed the title Experiment: Synced Collections optimizations Defer statepoint instantiation Feb 19, 2021
@bdice bdice changed the title Defer statepoint instantiation Defer statepoint instantiation, allow skip of SyncedCollection validation Feb 19, 2021
@bdice bdice force-pushed the synced_collections/lazy_statepoint_init branch from 5c32722 to 60534de Compare February 19, 2021 21:14
@vyasr vyasr mentioned this pull request Feb 19, 2021
12 tasks
@bdice bdice changed the title Defer statepoint instantiation, allow skip of SyncedCollection validation Defer statepoint instantiation, unify reset_statepoint logic Feb 19, 2021
@bdice bdice added the enhancement New feature or request label Feb 19, 2021
@bdice bdice marked this pull request as ready for review February 19, 2021 21:16
Copy link
Contributor

@vyasr vyasr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One comment, but not something that needs to be addressed before this is merged.

signac/contrib/job.py Show resolved Hide resolved
@bdice bdice requested a review from vyasr February 20, 2021 01:04
@bdice bdice merged commit e5b8057 into feature/synced_collections Feb 20, 2021
@bdice bdice deleted the synced_collections/lazy_statepoint_init branch February 20, 2021 01:36
vyasr added a commit that referenced this pull request Feb 20, 2021
* Improve Sync Data Structures (#336)

Adds a generic framework for synced collections to signac. The abstract SyncedCollection defines the interface for any Collection-like object (as defined by Python's collections.abc.Collection) that is synced with some form of persistent storage. This concept generalizes the existing _SyncedDict and SyncedList classes by implementing a common framework for these. Critically, this framework decouples the act of saving to and loading from the backend from the recursive conversion between a SyncedCollection and its base data type (e.g. a SyncedDict to a dict and vice versa), the latter of which is necessary to ensure proper synchronization of nested collections. This decoupling makes it much easier to enable different back ends for a given data structure without having to modify the synchronization logic.

* Add alternative backends for synced_collection (#364)

Adds Redis, Zarr, and MongoDB backends.

* Added validation layer to SyncedCollection (#378)

SyncedCollections can now arbitrary validators to input data to ensure that data conforms to a desired specification. This layer can be employed to ensure, for instance, that all inputs are valid JSON.

* Added buffering to SyncedCollection (#363)

Adds the ability to buffer I/O in SyncedCollections using an in-memory cache to store data.

* Feature/synced collection/reorg (#437)

Reorganizes synced collections into their own subpackage and move all corresponding tests into a subdirectory for easier identification.

* Feature/synced collection/reorg tests (#438)

Cleans up and standardizes the testing architecture so that all backends can easily apply the exact same set of tets using a few minor extra pieces of information. Prevents duplicate execution of tests upon import, while also adding a large number of additional tests for backends that were not previously tested.

* Fix incomplete merge

* Remove lingering old file.

* Feature/synced collection/cleanup (#445)

Rename various methods and remove unnecessary ones from the public API. Standardize internal APIs and simplify the implementation of new backends by automating more subclass behaviors. Improve constructors to enable positional arguments. Improve interfaces for various backends by making it easier for the user to specify and access the precise storage mechanisms.

* Merge master, apply all relevant formatting tools, and add documentation (#446)

Makes the SyncedCollection framework adhere to black, isort, flake8, pydocstyle, and mypy requirements while adding sufficiently detailed documentation to all classes.

* Feature/synced collection/cleanup2 (#447)

Simplifies and standardizes file buffering implementation. Adds extra tests for SyncedAttrDict.update and simplifies its implementation to use _update.

* Feature/synced collection/optimization (#453)

Optimize various aspects of the SyncedCollection framework, including validators, abstract base class type lookups, and the core data structures.

* Remove unnecessary backend str from tests.

* Feature/synced collection/test mongodb redis (#464)

MongoDB and Redis will no longer be silently skipped on CI, so any changes that break support for those will be immediately discovered.

* Make SyncedCollections thread-safe (#463)

Operations that modify SyncedCollections (and their subclasses) now acquire object-specific mutexes prior to modification in order to guarantee consistency between threads. Thread safety can be enabled or disabled by the user.

* Implements an in-memory buffer for SyncedCollections (#462)

The new buffering mode is a variant on the old one that avoids unnecessary encoding, decoding, and in-memory updating by directly sharing memory between various collections accessing the same file in the buffer. This direct sharing allows all changes to be automatically persisted, avoiding any cache inconsistencies without the high overhead of JSON serialization for every single modification.

* Clean up miscellaneous outstanding to-do items (#466)

Completes TODO items scattered throughout the code base and removes a number of outdated ones that have already been addressed.

* Make buffering thread safe (#468)

In addition to synced collections being thread safe individually, while in buffered mode the buffer accesses also have to be made thread safe for multithreaded operation to be safe. This pull request introduces sufficient locking mechanisms to support thread-safe reads from and writes to the buffers.

* Feature/synced collection/unify buffering (#469)

Unifies the implementation of the two different file buffering modes as much as possible using a shared base class. In addition, this fixes a couple of issues with the thread-safe buffering solution in #468 that only show up on PyPy where execution is fast enough to stress-test multithreaded execution. It also reduces thread locking scopes to the minimum possible.

* Feature/synced collection/contexts (#470)

Removes usage of contextlib.contextmanager and replaces it with custom context classes implementing the context manager protocol. The contextlib decorator has measurable overhead that these pre-instantiated context managers avoid. Furthermore, many of the context managers in synced collections follow a very similar counter-based pattern that is now generalized and shared between them.

* Install pymongo on pypy.

* Don't sync on construction.

* Add comparison operators to SyncedList and make sure modifying the filename of JSONDict is safe.

* Remove unnecessary constructor validation, providing both data and resource arguments (e.g. filename for a JSONDict) is valid.

* Fix unused imports.

* Feature/synced collection/replace jsondict (#472)

The old JSONDict and SyncedAttrDict classes are replaced with the new ones from the SyncedCollections framework. The new classes are now used for the Job's statepoint and document attributes as well as the Project document. The state point is now stored in the new _StatePointDict class, which extends the new JSONDict class to afford greater control over saving and loading of data. Much of the internals of the Job class have also been simplified, with most of the complex logic for job migration and validation when the state point changes now contained within the _StatePointDict.

* Replace old JSONDict with new BufferedJSONDict.

* Verify that replacing BufferedJSONDict with MemoryBufferedJSONDict.

* Remove largely redundant _reset_sp method.

* Remove single-use internal functions in Job to reduce surface area for SyncedCollection integration.

* Move logic from _init into init.

* Working implementation of statepoint using new SyncedCollection.

* Remove _check_manifest.

* Expose loading explicitly to remove the need for internal laziness in the StatepointDict.

* Simplify the code as much as possible by inlining move method and catching the correct error.

* Improve documentation of context manager for statepoint loading.

* Replace MemoryBufferedJSONDict in Project for now.

* Add documentation of why jobs must be stored as a list in the statepoint.

* Address PR comments.

* Add back import.

* Ensure _StatepointDict is always initialized in constructor.

* Change _StatepointDict to validate id on load.

* Refactor error handling into _StatepointDict class.

* Update docstrings.

* Update comment.

* Fix some docstrings.

* Remove redundant JobsCorruptedError check.

* Rewrite reset_statepoint to not depend on creating another job.

* Reduce direct accesses of internal attributes and do some simplification of the code.

* Reraise errors in JSONCollection.

* Change reset to require a non-None argument and to call _update internally.

* Add reset_data method to provide clear access points of the _StatepointDict for the Job.

* Create new internal method for handling resetting.

* Move statepoint resetting logic into the statepoint object itself.

* Stop accessing internal statepoint filename attribute directly and rely on validation on construction.

* Make statepoint thread safe.

* Some minor cleanup.

* Remove now unnecessary protection of the filename key.

* Explicitly document behavior of returning None from _load_from_resource.

* Apply suggestions from code review

Co-authored-by: Bradley Dice <[email protected]>

* Rename SCJSONEncoder to SyncedCollectionJSONEncoder.

* Only access old id once.

* Move lazy attribute initialization into one location.

* Address PR requests that don't cause any issues.

* Remove the temporary state point file backup.

* Make as many old buffer tests as possible.

* Reset buffer size after test.

* Last changes from PR review.

Co-authored-by: Bradley Dice <[email protected]>

* Fix synced collection support for 0d numpy arrays.

* Add oldest supported version of pymongo and ensure that zarr/mongo collections don't fail on import.

* Deprecate json module (#480)

Change all non-deprecated modules to import the built-in json module instead of signac's and deprecate signac.core.json.

* Feature/synced collection/reorg (#481)

Reorganizes the package structure so that the synced_collections subpackage is now at the package root and is internally structured with subpackages for data types, backends, and buffers.

* Move synced collections from core to package root.

* Fix all import locations.

* Reorganize internals of synced collection package.

* Fix all imports for reorganized package.

* Hide caching module since it's still experimental.

* Address PR comments.

* Make __all__ into an empty list not rather than a list containing an empty string.

* Feature/synced collection/simplify global buffering (#482)

Eliminate the global buffering mode across all backends in favor of a more targeted per-backend approach.

* Enable per backend buffering.

* Remove global buffering in favor of class-specific buffering.

* Reintroduce warnings for deprecated functionality.

* Remove truly global buffering and replace it with class-level buffering.

* Document new features.

* Feature/synced collection/deprecate old (#483)

Deprecates the old _SyncedDict, SyncedAttrDict, and JSONDict classes, along with any associated functions and exceptions.

* Deprecate old synced dict classes.

* Move class deprecation warnings to constructors.

* Feature/synced collection/fix buffer reload (#486)

* Make sure data type is preserved when reloading from buffer after flush.

* Add test of new error case.

* Fix lots of documentation issues.

* Address first round of PR comments.

* Update changelog.

* Fix mypy error.

* Don't error check uid unless it's provided.

* First pass to address PR comments.

* Feature/synced collection/remove attr access (#504)

This patch removes attribute-based access to synced dictionaries. This logic is moved to a new `AttrDict` mixin class that can be inherited by any other subclasses if this feature is desired.

* Simplify definition of __setattr__ by relying on complete list of protected keys.

* Move all attribute-based access to a separate mixin class AttrDict.

* Rename SyncedAttrDict to SyncedDict.

* Move synced_attr_dict to synced_dict.

* Remove attribute-based access from existing backend dict types and add new validator to check string keys.

* Isolate deprecated non-str key handling to the _convert_key_to_str json validator.

* Add tests of the AttrDict behavior.

* Use new attrdict based classes in signac and make all tests pass.

* Remove support for inheriting protected attributes, they must be defined by the class itself.

* Change initialization order to call super first wherever possible.

* Address PR comments.

* Address final round of PR comments.

* Feature/synced collection/numpy (#503)

Isolate all numpy logic into a single utility file so that handling of numpy arrays can be standardized. Also substantially improves test coverage, testing a large matrix of different numpy dtypes and shapes with different types of synced collections. The testing framework as a whole is refactored to simplify the code and reduce the amount of boilerplate required to add the new numpy tests.

* Initial pass at isolating all numpy logic to a single file.

* Use pytest to generate temporary directories.

* Stop saving backend kwargs as class variables.

* Remove most references to class-level variables in tests.

* Remove the _backend_collection variable.

* Remove unnecessary autouse fixtures.

* Start adding comprehensive tests for numpy conversion.

* Make sure locks are released even if saving fails.

* Add tests of vector types and add tests for SyncedList as well as SyncedDict.

* Use pytest to parametrize numpy types instead of looping manually.

* Unify vector and scalar tests.

* Stop testing squeezing and just test the relevant shapes directly.

* Add test of reset.

* Move numpy tests back to main file.

* Remove _conert_numpy_scalar, which performed undesirable squeezing of 1-element 1d arrays, and replace usage with _convert_numpy.

* Separate numpy testing into separate functions and limit supported behavior to only the necessary use cases.

* Add a warning when implicit numpy conversion occurs.

* Update changelog.

* Address all PR comments aside from numpy conversion in type resolver.

* Catch numpy warnings in signac numpy integration tests.

* Support older versions of numpy for random number generation.

* Fix isort issue introduced by rebase.

* Address PR comments.

* Allow AbstractTypeResolver to perform arbitrary preprocessing, delegating the numpy-specialized code to the caller and making it less confusing.

* Add missing call to _convert_numpy.

* Set NUMPY_SHAPES for MongoDB tests when numpy is not present.

* Remove add_validators and specify validators in class definition (#507)

* Remove add_validators classmethod and instead require validators to be defined at class definition time, preventing one application from modifying validation process for others and giving a standard means to completely override parent validators in parents.

* Change type in docstring.

* Don't use os.path.join where not needed. (#511)

The extra work performed by os.path.join can be slow, so this PR replaces it with direct string concatenation of os.sep.

* Don't use os.path.join where not needed.

* Update signac/contrib/job.py

Co-authored-by: Bradley Dice <[email protected]>

* Disable recursive validation during recursive conversion of nested types. (#509)

* Feature/synced collection/optimize jsondict validation (#508)

Define a single validator for JSONAttrDict classes that combines the logic of other validators while reducing collection traversal costs. Also switch from converting numpy arrays to just bypassing the resolver's cache for them.

* Use single separate validator for state points for performance.

* Remove preprocessor from type resolver and instead use a blocklist that prevents caching data for certain types.

* Reorder resolvers to optimize performance.

* Make sure not to include strings as sequences.

* Move state point validator to collection_json and use for all JSONAttrDict types.

* Make sure to also check complex types.

* Add back missing period lost during stash merge.

* Address review comments.

* Feature/synced collection/optimize protected key lookup (#510)

Since the most common operation on protected keys is to check if some key is within the list of protected keys, this patch changes the `_PROTECTED_KEYS` attribute to a set for faster O(1) membership checks.

* Switch protected keys from a sequence to a set for faster containment checks.

* Change evaluation order of checks in setattr.

* Address PR comments.

* Defer statepoint instantiation, unify reset_statepoint logic (#497)

* Defer state point initialization when lazy loading.

* Allow validation to be disabled in SyncedCollection._update.

* Unify reset_statepoint logic across methods.

* Revert validation-related changes.

* Add test, fix bug.

* Update tests/test_job.py

* Feature/synced collection/optimize (#513)

* Add a resoler to fast-exit _from_base for non-collection types.

* Optimize synced collection init method.

* Remove validators property and access internal variable directly.

* Add early exit for _convert_array.

* Use locally cached root var instead of self._root.

* Remove superfluous duplicate check.

* Optionally skip validation in SyncedCollection _update. (#512)

* Add option to trust the data source during _update.

* Skip validation if JSON is valid.

* Fix pre-commit.

* Rename trust_source to _validate.

* Set _validate=False.

Co-authored-by: Vishav Sharma <[email protected]>
Co-authored-by: Bradley Dice <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants