Searching Across Multiple Indexes

[a-drew]

Would anyone be interested in this package supporting searches across multiple indexes / model types? I feel this is a significant change so instead of opening up a PR, I wanted to see if this is something that would benefit anyone other than myself.

I recently found this package and started integrating it into one of my projects, however I have a few peculiar requirements that I had to work through:

• Support multiple localizations, aka 1 index per locale
• Provide an All Results page which displays results from different model types

The first one was easy, I simply made searchableAs return a different index depending on the current app locale.
But the second one is where things got tricky..

Implementation

The easiest approach would've been to just search once per model, merge the results and sort by score once again. However, I know that elasticsearch does support searching against multiple indexes at the same time, So I decided to reduce the amount of queries sent to elasticsearch and have it sort everything for me.

I know that elastic-scout-driver-plus has this sort of join feature but I require some flexibility in terms of elasticsearch version. That package won't let me swap away from ES 8 and might make it impossible to move to opensearch down the line.

You can find a proof of concept on my fork (https://github.com/a-drew/Explorer/tree/search-across-multiple-indexes) (diff here)

• Definitely inspired by elastic-scout-driver-plus, I've added some generic classes for representing Hits, Documents and Highlights and modified the Results class to let me easily grab collections of hits, documents or even models.
• Each Hit is responsible for figuring out its own model.
• I've added a getAliases method onto the ElasticIndexAdapter so that an IndexConfiguration could be identified from its aliases.
• I've updated ElasticEngine::map to support mapping to any model type, this could be further improved to reduce the amount of db queries.
• Builder now has 2 new methods join and joinMany, these accept model instances or class names and will update the ES query to search across all the indices for the provided models.
• Builder now overrides paginate and paginateRaw returning a custom LengthAwarePaginator that can quickly swap between different search representations.
• Hit class is resolved from the container so you can swap it out for your own subclass modify it to your liking.

Usage

Ex: Finding Articles and Authors that contain the word john

$models = \App\Models\Article::search('john')->join(\App\Models\Author::class)->get();

Ex: Paginated results containing Articles, Authors and Videos

$query = \App\Models\Article::search('will')->joinMany([\App\Models\Author::class, \App\Models\Video]);

$paginatedModels = $query->paginate(); // still defaults to a Models collection

$paginator = $query->paginateRaw(); // now contains a collection of `Hit` items instead of one `Results` object

$paginator->asDocuments(); // collection is now swapped to `Document` items

$paginator->asModels() // collection is now swapped to `Model` items

$paginator->asHits() // collection is now swapped back to `Hit` items

All other query methods work as expected

$models = \App\Models\Article::search('dancing queen')->join(\App\Models\Video::class)
              ->should(new Term('artist', 'ABBA', 10))->get();

[Jeroen-G]

Wow, that's quite some work you put into this idea. I'll study it and come back to you!

[blackshadev]

I like the idea a lot! But for the implementation... I am struggling.

First the things I like:

• An valueObject for Hit , Document and Highlight. That is something I did miss, but it would also warrent a major breaking change, not the end of the world, just something to keep in mind. As long as it isn't fully backwards compatible, this will need some time to release.
• A own paginator, always great
• The custom builder ❤️ if you opened a seperate PR for this, this would be very nice for PHP 8.2

Than what I think you can improve upon:

• I'd like to have an approach which works in all cases, instead of creating an "if" condition in src/Infrastructure/Scout/ElasticEngine.php to check if it has multiple models.
  • If this is not possible, move the logic for checking if there are more indices to the builder, the mapping itself can stay
• The liberal use of the container from the Results and Hit class. I would like to have these are "ValueObjects" which are immutable and stateless. By the container this way (instead of dependency injection), it becomes really hard to track method usage, but also some instance usage. Which makes it harder for maintainers to maintain stability over multiple versions. It also makes it harder to test without bootstrapping the container, which is not really what we'd want. Ideally we can test these new value objects (Document, Hit, Highlight) without bootstrapping or faking the container in any way. This makes tests easier to write, run and maintain.