RFC: tile pipelines as "virtual tile sources"

[MichaelKreil]

Introduction

VersaTiles should support "virtual tile sources" defined via YAML, allowing users to create flexible and powerful tile processing pipelines. In addition to standard container formats such as *.versatiles, *.mbtiles, *.pmtiles, ... virtual tile sources *.yaml will allow for dynamic tile processing and transformation workflows. These pipelines can be used as virtual tile sources for tile conversion, but can also be used as a source for serving map tiles.

Use cases

• Tile conversion
  • Render vector tiles as image tiles
  • Tracing image tiles to vector tiles
  • Converting between tile formats
• Editing image tiles
  • Recolour tiles
  • Changing resolution
• Editing vector tiles
  • Delete layers
  • Adding layers from other sources
  • Update/remove properties of features
• Merge multiple tile sources
  • Combine zoom levels from different sources, e.g. use zoom levels 1-12 from one source and zoom levels 13-14 from another.
  • Overlay image tiles, e.g. overlay aerial imagery over satellite tiles to create a composite view.

Example 1: Appending new properties to POI features

This example shows how to open a vector tile containing Points of Interest (POIs) and append new properties to each POI feature by matching the 'id' field from a CSV file.

- run: read
  args:
    filename: pois.versatiles

- run: vectortiles_update_properties
  args:
    data_source_path: pois.csv
    id_field_tiles: id
    id_field_values: id

Example 2: Adding a GeoJSON as a layer to vector tiles

This example shows how to read vector tiles from europe.versatiles and add a new layer called "overlay" containing geometries from overlay.geojson.

- run: vectortiles_merge_layer
  sources:
    - run: read
      args:
        filename: europe.versatiles
    
    - run: read_geojson
      args:
        layer_name: overlay
        max_zoom: 8
        filename: overlay.geojson

Specifications

Supported operations

1. Reader: Read tiles from a tile container or create tiles from a geometry or image file.
2. Transformer: transforms tiles.
3. Composer: Merge/join/overlay multiple tile sources into a single tile source.

Pipeline Definition

A pipeline is defined as a sequence of operations, where each operation specifies:

• The operation to be performed (run).
• The arguments required by the operation (args).
• The sources to be used (source) - only for Composers.

Documentation

If you run versatiles help pipeline you get a documentation of all operation, e.g.:

/cc @yetzt

[yetzt]

merging different sources looks important to me, so far i had to convert to tar, use tar --concatenate, convert back to versatiles.

the editing vectors and rasters is what tileblaster can do, and it can be a pretty powerful tool. i don't know if defining complex operations with yaml makes the most sense. i feels this would require a more expressive and less problem-ridden document language like kdl or an actual interpreter like lua or some embedded js interpreter, at least as an advanced option.

[MichaelKreil]

Thank you! KDL looks great and might actually be better for this use case.
And yes, I'm trying not to add a LUA or ejs interpreter. That would feel like a "solution" that creates 5 new problems.

[yetzt]

maybe boa in the future? :)

[MichaelKreil]

After much experimentation, I have reinvented the wheel! \o/

But seriously, when defining a pipeline, you basically have two types of lists. A list of things like multiple values or multiple tile sources, and a list of operations that define a pipeline. In the first case (list of values/sources) it would be great to have a notation like: [source1, source2, source3]. For the second case (pipeline) it would be great to be able to use a pipe notation: operation1 | operation2 | operation3.
A second point is that it would be nice if whitespace (like line breaks) had no semantic meaning. This would make it easier to define a pipeline as a inline bash argument.
So in the end I came up with a solution based on KDL, but more suitable for our use cases.
To give you an overview, I paste the automatically generated Markdown help here:

---

VersaTiles Pipeline

VersaTiles Pipeline is a robust toolkit designed for efficiently generating and processing large volumes of tiles, from thousands to millions. It leverages multithreading to stream, process, and transform tiles from one or more sources in parallel, either storing them in a new tile container or delivering them in real-time through a server:

# save the processed tiles in a container:
versatiles convert pipeline.vpl result.versatiles

# serve the tiles directy via the server:
versatiles serve pipeline.vpl

Defining a pipeline

To define a pipeline, create a .vpl file and descibe the pipeline using the VersaTiles Pipeline Language (VPL). Pipelines always begin with a read operation, optionally followed by one or more transform operations, separated by the pipe symbol (|).

Example:

get_tiles filename="world.versatiles" | do_some_filtering | do_some_processing

Operation Format

Each operation follows this structure:

operation_name parameter1="value1" parameter2="value2" ...

For read operations that combine multiple sources, use a comma-separated list within square brackets:

Example:

get_overlayed [
   get_tiles filename="world.versatiles",
   get_tiles filename="europe.versatiles" | do_some_filtering,
   get_tiles filename="germany.versatiles"
]

---

READ operations

get_dummy_tiles

Generates mocked tiles.

Parameters:

• format: String (optional) - Specifies the tile format.
• compression: String (optional) - Compression type.
• min_zoom: u8 (optional) - Minimum zoom level.
• max_zoom: u8 (optional) - Maximum zoom level.
• min_lat: f32 (optional) - Minimum latitude.
• max_lat: f32 (optional) - Maximum latitude.
• min_lng: f32 (optional) - Minimum longitude.
• max_lng: f32 (optional) - Maximum longitude.

get_overlayed

Overlays multiple tile sources, using the tile from the first source that provides it.

Sources:

• All tile sources must have the same format.

get_tiles

Reads a tile source, such as a VersaTiles container.

Parameters:

• filename: String (required) - The filename of the tile container, e.g., "world.versatiles".

---

TRANSFORM operations

vectortiles_update_properties

Updates properties of vector tile features using data from an external source (e.g., CSV file). Matches features based on an ID field.

Parameters:

• data_source_path: String (required) - Path to the data source file, e.g., data.csv.
• id_field_tiles: String (required) - ID field name in the vector tiles.
• id_field_data: String (required) - ID field name in the data source.
• layer_name: String (optional) - Name of the layer to update. If unspecified, all layers will be updated.
• replace_properties: Boolean (optional, default: false) - If set, old properties will be deleted before new ones are added.
• remove_empty_properties: Boolean (optional, default: false) - If set, features with no properties will be removed.
• include_id: Boolean (optional, default: false) - If set, includes the ID field in the updated properties.

[sastcsgh]

Yesterday, I had a hunch this could be useful and switched our base tilesets into a .vpl file. I still need to do some more testing, but it seems great so far. Now I can pretend our three base sources are actually one file, and it greatly helps with version control as well. Now I don't have to keep around an old osm file just in case some older json style file someone has cached still refers to it - rather I can just replace the file and only keep the old .vpl file around which now refers to the new one automatically, so I can name my tilesets whatever I want while keeping the file names server side as simple as possible. Very happy 😄

[MichaelKreil]

I'm also very happy with VPL. But there is still a lot to be done. If you have any ideas for improvements or feature requests, just add an issue or pull request to versatiles-rs.