Query for Spark Integration : Lesson 3 - Fast Nearest Neighbours

[rbhatia46]

Hi,
This is with respect to - /source/notebooks/L3/nearest-neighbor-faster.ipynb.
Really liked the idea and thanks for the example, this is really useful. However, I was wondering can this be parallelized using Spark to achieve a further speedup, I have 10s of billions of rows, and using Spark currently to process all of the lat-long data, and I have another Dataframe, which has 10 Million Points of Interest, I want to perform a nearest neighbor analysis to find closest Point of Interest.

I am already performing clustering to reduce the dataset size, but it would be really interesting to see if this can be further integrated with Spark to achieve further latency reduction.

Look forward to your thoughts.