README.md

Ship2Ship Transfer Detection

Note: ipynb files can be previewed in GitHub and can also be imported into Databricks, more here.

Introduction

This is an algorithmic implementation to detect Ship to Ship transfers at scale using Databricks. It was presented at the Data and AI Summit 2022.

This Mosaic example explores a novel, algorithmic approach to detecting Ship to Ship transfers at scale using AIS data. In particular it aims to surpass existing, naive implementations that are just based on a particular distance radius like the one shown below:

Although the naive approach can be optimised with indices to be quite performant, additional improvements can be made. This is quite apparent if we look at the following data points below:

According to our naive approach, where we buffer around our LAT/LONG points, the two vessels would not intersect. However, if we construct the actual path the vessels took, our algorithmic implementation would detect an overlap between the two paths, as shown below:

This model is expanded upon in the course of the attached notebooks. It shows how to ingest AIS data, how to process it at scale leveraging Mosaic in Databricks, and provides examples of how to extend the analysis to incorporate additional sources.

Requirements

This notebook series was run on Databricks with the Mosaic library installed. For this example it is recommended to install it from PyPi on your Databricks cluster like so.

created by @tiems90