Skip to content

Latest commit

 

History

History
181 lines (121 loc) · 12.4 KB

README.md

File metadata and controls

181 lines (121 loc) · 12.4 KB
Raw

Mosaic by Databricks Labs

mosaic-logo

An extension to the Apache Spark framework that allows easy and fast processing of very large geospatial datasets.

PyPI version PyPI - Downloads codecov build docs Code style: black lines of code

Why Mosaic?

Mosaic was created to simplify the implementation of scalable geospatial data pipelines by bounding together common Open Source geospatial libraries via Apache Spark, with a set of examples and best practices for common geospatial use cases.

What does it provide?

Mosaic provides geospatial tools for

mosaic-general-pipeline

The supported languages are Scala, Python, R, and SQL.

How does it work?

The Mosaic library is written in Scala (JVM) to guarantee maximum performance with Spark and when possible, it uses code generation to give an extra performance boost.

The other supported languages (Python, R and SQL) are thin wrappers around the Scala (JVM) code.

mosaic-logical-design Image1: Mosaic logical design.

Getting started

⚠️ geopandas 0.14.4 not supported

For Mosaic <= 0.4.1 %pip install databricks-mosaic will no longer install "as-is" in DBRs due to the fact that Mosaic left geopandas unpinned in those versions. With geopandas 0.14.4, numpy dependency conflicts with the limits of scikit-learn in DBRs. The workaround is %pip install geopandas==0.14.3 databricks-mosaic. Mosaic 0.4.2+ limits the geopandas version.

Mosaic 0.4.x Series [Latest]

We recommend using Databricks Runtime versions 13.3 LTS with Photon enabled.

⚠️ Mosaic 0.4.x series only supports DBR 13. If running on a different DBR with throw an exception:

DEPRECATION ERROR: Mosaic v0.4.x series only supports Databricks Runtime 13. You can specify %pip install 'databricks-mosaic<0.4,>=0.3' for DBR < 13.

⚠️ Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster [ADB | AWS | GCP]:

DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.

Language Bindings

As of Mosaic 0.4.0 / DBR 13.3 LTS (subject to change in follow-on releases)...

  • Assigned Clusters
    • Mosaic Python, SQL, R, and Scala APIs.
  • Shared Access Clusters
    • Mosaic Scala API (JVM) with Admin allowlisting.
    • Mosaic Python bindings (to Mosaic Scala APIs) are blocked by Py4J Security on Shared Access Clusters.
    • Mosaic SQL expressions cannot yet be registered with Unity Catalog due to API changes affecting DBRs >= 13, more here.

Additional Notes:

Mosaic is a custom JVM library that extends spark, which has the following implications in DBR 13.3 LTS:

  1. Unity Catalog: Enforces process isolation which is difficult to accomplish with custom JVM libraries; as such only built-in (aka platform provided) JVM APIs can be invoked from other supported languages in Shared Access Clusters.
  2. Volumes: Along the same principle of isolation, clusters can read Volumes via relevant built-in (aka platform provided) readers and writers or via custom python calls which do not involve any custom JVM code.

Mosaic 0.3.x Series

We recommend using Databricks Runtime versions 12.2 LTS with Photon enabled.

⚠️ Mosaic 0.3.x series does not support DBR 13.

As of the 0.3.11 release, Mosaic issues the following WARNING when initialized on a cluster that is neither Photon Runtime nor Databricks Runtime ML [ADB | AWS | GCP]:

DEPRECATION WARNING: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic will stop working on this cluster after v0.3.x.

If you are receiving this warning in v0.3.11+, you will want to begin to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic has standardized to JTS as its default and supported Vector Geometry Provider.

Documentation

Check out the documentation pages.

Python

Install databricks-mosaic as a cluster library, or run from a Databricks notebook

%pip install databricks-mosaic

Then enable Mosaic (and namespace it in python) with

import mosaic as mos
mos.enable_mosaic(spark, dbutils)

Scala

Get the jar from the releases page and install it as a cluster library.

Then enable it with

import com.databricks.labs.mosaic.functions.MosaicContext
import com.databricks.labs.mosaic.H3
import com.databricks.labs.mosaic.JTS

val mosaicContext = MosaicContext.build(H3, JTS)
import mosaicContext.functions._

R

Get the Scala JAR and the R from the releases page. Install the JAR as a cluster library, and copy the sparkrMosaic.tar.gz to DBFS (This example uses /FileStore location, but you can put it anywhere on DBFS).

library(SparkR)

install.packages('/FileStore/sparkrMosaic.tar.gz', repos=NULL)

Enable the R bindings

library(sparkrMosaic)
enableMosaic()

SQL

Configure the Automatic SQL Registration or follow the Scala installation process and register the Mosaic SQL functions in your SparkSession from a Scala notebook cell:

%scala
import com.databricks.labs.mosaic.functions.MosaicContext
import com.databricks.labs.mosaic.H3
import com.databricks.labs.mosaic.JTS

val mosaicContext = MosaicContext.build(H3, JTS)
mosaicContext.register(spark)

Note: Mosaic 0.4.x SQL bindings for DBR 13 can register with Assigned clusters (as Spark Expressions), but not Shared Access due to API changes, more here.

Examples

Here are some example notebooks, check the language links for latest [Python | Scala | SQL | R]:

Example Description Links
Quick Start Example of performing spatial point-in-polygon joins on the NYC Taxi dataset python, scala, R, SQL
Shapefiles Examples of reading multiple shapefiles python
Spatial KNN Runnable notebook-based example using Mosaic SpatialKNN model python
NetCDF Read multiple NetCDFs, process through various data engineering steps before analyzing and rendering python
STS Transfers Detecting Ship-to-Ship transfers at scale by leveraging Mosaic to process AIS data. python, blog
EO Gridded STAC End-to-end Earth Observation series showing downloading Sentinel-2 STAC assets for Alaska from MSFT Planetary Computer, tiling them to H3 global grid, band stacking, NDVI, merging (mosaicing), clipping, and applying a Segment Anything Model python

You can import those examples in Databricks workspace using these instructions.

Ecosystem

Mosaic is intended to augment the existing system and unlock the potential by integrating Spark, Delta Lake and 3rd party frameworks into the Lakehouse architecture.

mosaic-logo Image2: Mosaic ecosystem - Lakehouse integration.

Project Support

Please note that all projects in the databrickslabs github space are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects.

Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support.