Skyplane UDFs

[sarahwooders]

Some users have requested adding additional flexibility to Skyplane to enable custom object processing during transfers. We propose supporting with via two features: 1. custom transformations (i.e. UDFs) and 2. custom object chunking.

Custom Transformations

We can support UDFs to be run by Skyplane gateways, either in the source region VMs or destination region VMs. These UDFs must be stateless to simplify parallelization of chunk processing on VMs.

def encrypt(chunk_data: bytes) -> bytes: 
    pass

def decrypt(chunk_data: bytes) -> bytes: 
    pass
 
client = SkyplaneClient()   
client
     .copy("s3://src-bucket/", "gs://dst-bucket")
     .on_download(encrypt)
     .on_upload(decrypt)

Custom Line Chunker

Users can customize how to convert objects into chunks.

Definition:

class CustomChunker(Chunker): 

    def chunkify(self, obj: ObjectStoreObject) -> List[Chunk]: 
         # return list of chunks that specify the byterange, chunk_size, partition, key, and byte offset 
        pass

Usage:

client = SkyplaneClient()
client.copy("s3://src-bucket/", "gs://dst-bucket", chunker=CustomChunker())

Custom Object Store Interface

Some users may want destination data to remain in chunk format, rather than reconstruct the object. This can be accomplished by writing a custom ObjectStoreInterface implementation (an example for S3 below):

class ChunkedDestinationInterface(S3Interface): 

     def upload_object(
          self, src_file_path, dst_object_name, part_number=None, upload_id=None, check_md5=None, mime_type=None
    ):
        key = os.path.basename(src_file_path) # use chunk ID instead of original object key 
        s3_client.put_object(Body=f, Key=key, Bucket=self.bucket_name, **checksum_args, **mime_args)
    
    def complete_multipart_upload(self, key: str, upload_id: str): 
        # don't need to complete multipart uploads, since we want data to remain as chunks 
       return 

Usage:

client = SkyplaneClient()
client.copy("s3://src-bucket/", "gs://dst-bucket", dest_iface = ChunkedDestinationInterface())

[jegonzal]

Would these UDFs be stateful?

[salieri11]

Hi,
What is the definition of Object? Sky plane moves bytes, however, defining Object in any way sounds less generic to me