Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speeding performance on EntryType instance and Connection add #4

Open
TcaManager opened this issue Sep 4, 2019 · 2 comments
Open
Labels
enhancement New feature or request

Comments

@TcaManager
Copy link

Hi,
It's not a real issue, we use your Framework to make validation of entries before exporting an ldif. With a code that looks like this

ldif_connection = Connection(server=None,client_strategy='LDIF')
user_entry_type = EntryType('...') #Put whatever data here, dn is a template using uid attribute
for user_data in users_data: # users_data is a list of dict containing more than 10 attributes
    user = user_entry_type(user_data) # It takes about 0.001 for each user
    ldif_connection.add( # This too takes about 0.001 for each user
        user.entry_dn,
        user.object_classes,
        user.entry_attributes_as_dict
    )

We found that each of the two operations in the loop cost 0.001 s.
Are there some parameters we could use to speed up one of those operations?
Is it possible to make it much faster with more CPU or RAM?
Can multprocessing help here?
Thanks.

@TcaManager
Copy link
Author

For example, if I profile a script inserting 2 users, I get the following data with the Spyder Code Profiler,
image

@cfelder
Copy link
Owner

cfelder commented Sep 5, 2019

Thanks for reaching out @TcaManager. Indeed there is some overhead on instantiation of dynamic classes. I'll have a look at this but it will need some time to profile the code myself to pin point the problem. If you are using SMP multiprocessing can increase performance by chunking your entries to several processes. Nevertheless this just mitigates the real issue.

@cfelder cfelder added the enhancement New feature or request label Oct 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants