Sparkle support for Azure deployment

[kkiani]

Introduction

To extend Sparkle application support to Azure, we propose implementing the Resource Layer using Pulumi for deploying an Azure Synapse serverless PySpark application. This will allow Sparkle to run natively on Azure, leveraging Azure Synapse Analytics for distributed data processing with PySpark.

Tasks

• Implement Pulumi templates for deploying Azure Synapse resources.
• Define a Pulumi Component to provision the necessary components, including Azure Synapse workspaces, Spark pools, and necessary data storage.
• Ensure the deployment supports a serverless PySpark environment.
• Integrate this resource layer with Sparkle, maintaining consistency with the overall ARC pattern.
• Investigating implementation of getting the spark session inside Azure Synapse for controller layer.

Acceptance Criteria

• Successfully deploy a serverless Sparkle-based PySpark application on Azure Synapse via Pulumi.
• Validate that the Sparkle application runs and interacts seamlessly with Azure Synapse resources.

[kkiani]

We already have a bigger feature branch for integration with Sparkle. Please fork and create pull request to the feature/spark branch.