Performance/scalability issue in Folder::searchByMime (et al)

[paulijar]

The execution time of the function Folder::searchByMime() depends on the number of matching files in the whole user storage. That is, the matching files don't have to be within the targeted folder, just having them anywhere in the user storage slows down the search in all folders.

I ran into this when investigating the issue owncloud/music#664. There, the user had some 1.5 million image files (including the generated thumbnails) in the storage and running Folder::searchByMime('image') took 10 minutes even on empty folder!

The reason for this behavior is in the function Folder::searchCommon() in https://github.com/owncloud/core/blob/master/lib/private/Files/Node/Folder.php#L237. There, the function first fetches all the matching files in the whole storage and only then filters the result set according the target path. So in our example case, 1.5 million rows were retrieved from the DB table oc_filecache, a new CacheEntry object was instantiated for each row, and then all the 1.5 million objects were discarded.

As I see it, the filtering by path could as well happen already when making the SQL query in https://github.com/owncloud/core/blob/master/lib/private/Files/Cache/Cache.php#L642. That should be at least an order of magnitude faster than filtering afterwards in PHP foreach loop.

This is possibly also related to an old stalled issue #10077 although that seems to be more about using the search functions on shared folders, while the problem reported here doesn't need any shares to reproduce.

[ownclouders]

GitMate.io thinks the contributor most likely able to help you is @PVince81.

Possibly related issues are #21767 (issue), #4690 (Shared folder Issue), #6428 (Folder creation issue.), #4120 (Permission issues in shared folder), and #565 (Huge folders - Picture view issue).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.