Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add databricks_function resource #4189

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
112 changes: 112 additions & 0 deletions docs/resources/function.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
subcategory: "Unity Catalog"
---
# databricks_function Resource

-> This resource source can only be used with a workspace-level provider.

Creates a [User-Defined Function (UDF)](https://docs.databricks.com/en/udf/unity-catalog.html) in Unity Catalog. UDFs can be defined using SQL, or external languages (e.g., Python) and are stored within [Unity Catalog schemas](../resources/schema.md).

## Example Usage

### SQL-based function:

```hcl
resource "databricks_catalog" "sandbox" {
name = "sandbox_example"
comment = "Catalog managed by Terraform"
}

resource "databricks_schema" "functions" {
catalog_name = databricks_catalog.sandbox.name
name = "functions_example"
comment = "Schema managed by Terraform"
}

resource "databricks_function" "calculate_bmi" {
name = "calculate_bmi"
catalog_name = databricks_catalog.sandbox.name
schema_name = databricks_schema.functions.name
input_params = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be easier for people to specify input_param as separate blocks, like we do in other resources

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although doc says that input_params is a block, and then there are parameters inside: https://docs.databricks.com/api/workspace/functions/create#input_params

{
name = "weight"
type = "DOUBLE"
},
{
name = "height"
type = "DOUBLE"
}
]
data_type = "DOUBLE"
routine_body = "SQL"
routine_definition = "weight / (height * height)"
language = "SQL"
is_deterministic = true
sql_data_access = "CONTAINS_SQL"
security_type = "DEFINER"
}
```

### Python-based function:

```hcl
resource "databricks_function" "calculate_bmi_py" {
name = "calculate_bmi_py"
catalog_name = databricks_catalog.sandbox.name
schema_name = databricks_schema.functions.name
input_params = [
{
name = "weight_kg"
type = "DOUBLE"
},
{
name = "height_m"
type = "DOUBLE"
}
]
data_type = "DOUBLE"
routine_body = "EXTERNAL"
routine_definition = "return weight_kg / (height_m ** 2)"
language = "Python"
is_deterministic = false
sql_data_access = "NO_SQL"
security_type = "DEFINER"
}
```

## Argument Reference

The following arguments are supported:

* `name` - (Required) The name of the function.
* `catalog_name` - (Required) The name of the parent [databricks_catalog](../resources/catalog.md).
* `schema_name` - (Required) The name of [databricks_schema](../resources/schema.md) where the function will reside.
* `input_params` - (Required) A list of objects specifying the input parameters for the function.
* `name` - (Required) The name of the parameter.
* `type` - (Required) The data type of the parameter (e.g., `DOUBLE`, `INT`, etc.).
* `data_type` - (Required) The return data type of the function (e.g., `DOUBLE`).
* `routine_body` - (Required) Specifies the body type of the function, either `SQL` for SQL-based functions or `EXTERNAL` for functions in external languages.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see return_params

* `routine_definition` - (Required) The actual definition of the function, expressed in SQL or the specified external language.
* `language` - (Required) The language of the function, e.g., `SQL` or `Python`.
* `is_deterministic`- (Optional, `bool`) Whether the function is deterministic. Default is `true`.
* `sql_data_Access`- (Optional) The SQL data access level for the function. Possible values are:
* `CONTAINS_SQL` - The function contains SQL statements.
* `READS_SQL_DATA` - The function reads SQL data but does not modify it.
* `NO_SQL` - The function does not contain SQL.
* `security_type` - (Optional) The security type of the function, generally `DEFINER`.

## Attribute Reference

In addition to all arguments above, the following attributes are exported:
* `full_name` - Full name of the function in the form of `catalog_name.schema_name.function_name`.
* `created_at` - The time when this function was created, in epoch milliseconds.
* `created_by` - The username of the function's creator.
* `updated_at` - The time when this function was last updated, in epoch milliseconds.
* `updated_by` - The username of the last user to modify the function.

## Related Resources

The following resources are used in the same context:

* [databricks_schema](./schema.md) to get information about a single schema
* Data source [databricks_functions](../data-sources/functions.md) to get a list of functions under a specified location.
1 change: 1 addition & 0 deletions internal/providers/pluginfw/pluginfw_rollout_utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ var migratedDataSources = []func() datasource.DataSource{
var pluginFwOnlyResources = []func() resource.Resource{
// TODO Add resources here
sharing.ResourceShare, // Using the staging name (with pluginframework suffix)
catalog.ResourceFunction,
}

// List of data sources that have been onboarded to the plugin framework - not migrated from sdkv2.
Expand Down
226 changes: 226 additions & 0 deletions internal/providers/pluginfw/resources/catalog/resource_function.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
package catalog

import (
"context"
"fmt"
"time"

"github.com/databricks/databricks-sdk-go"
"github.com/databricks/databricks-sdk-go/apierr"
"github.com/databricks/databricks-sdk-go/retries"
"github.com/databricks/databricks-sdk-go/service/catalog"
"github.com/databricks/terraform-provider-databricks/common"
pluginfwcommon "github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/common"
pluginfwcontext "github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/context"
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/converters"
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/tfschema"
"github.com/databricks/terraform-provider-databricks/internal/service/catalog_tf"
"github.com/hashicorp/terraform-plugin-framework/diag"
"github.com/hashicorp/terraform-plugin-framework/path"
"github.com/hashicorp/terraform-plugin-framework/resource"
"github.com/hashicorp/terraform-plugin-framework/resource/schema"
)

const resourceName = "function"

var _ resource.ResourceWithConfigure = &FunctionResource{}

func ResourceFunction() resource.Resource {
return &FunctionResource{}
}

func waitForFunction(ctx context.Context, w *databricks.WorkspaceClient, funcInfo *catalog.FunctionInfo) diag.Diagnostics {
const timeout = 5 * time.Minute

result, err := retries.Poll[catalog.FunctionInfo](ctx, timeout, func() (*catalog.FunctionInfo, *retries.Err) {
attempt, err := w.Functions.GetByName(ctx, funcInfo.FullName)
if err != nil {
if apierr.IsMissing(err) {
return nil, retries.Continue(fmt.Errorf("function %s is not yet available", funcInfo.FullName))
}
return nil, retries.Halt(fmt.Errorf("failed to get function: %s", err))
}
return attempt, nil
})

if err != nil {
return diag.Diagnostics{diag.NewErrorDiagnostic("failed to create function", err.Error())}
}

*funcInfo = *result
return nil
}

type FunctionResource struct {
Client *common.DatabricksClient
}

func (r *FunctionResource) Metadata(ctx context.Context, req resource.MetadataRequest, resp *resource.MetadataResponse) {
resp.TypeName = pluginfwcommon.GetDatabricksProductionName(resourceName)
}

func (r *FunctionResource) Schema(ctx context.Context, req resource.SchemaRequest, resp *resource.SchemaResponse) {
attrs, blocks := tfschema.ResourceStructToSchemaMap(catalog_tf.FunctionInfo{}, func(c tfschema.CustomizableSchema) tfschema.CustomizableSchema {
c.SetRequired("name")
c.SetRequired("catalog_name")
c.SetRequired("schema_name")
c.SetRequired("input_params")
c.SetRequired("data_type")
c.SetRequired("routine_body")
c.SetRequired("routine_defintion")
c.SetRequired("language")

c.SetReadOnly("full_name")
c.SetReadOnly("created_at")
c.SetReadOnly("created_by")
c.SetReadOnly("updated_at")
c.SetReadOnly("updated_by")

return c
})

resp.Schema = schema.Schema{
Description: "Terraform schema for Databricks Function",
Attributes: attrs,
Blocks: blocks,
}
}

func (r *FunctionResource) Configure(ctx context.Context, req resource.ConfigureRequest, resp *resource.ConfigureResponse) {
if r.Client == nil && req.ProviderData != nil {
r.Client = pluginfwcommon.ConfigureResource(req, resp)
}
}

func (r *FunctionResource) ImportState(ctx context.Context, req resource.ImportStateRequest, resp *resource.ImportStateResponse) {
resource.ImportStatePassthroughID(ctx, path.Root("full_name"), req, resp)
}

func (r *FunctionResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
ctx = pluginfwcontext.SetUserAgentInResourceContext(ctx, resourceName)
w, diags := r.Client.GetWorkspaceClient()
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}

var planFunc catalog_tf.FunctionInfo
resp.Diagnostics.Append(req.Plan.Get(ctx, &planFunc)...)
if resp.Diagnostics.HasError() {
return
}

var createReq catalog.CreateFunctionRequest

resp.Diagnostics.Append(converters.TfSdkToGoSdkStruct(ctx, planFunc, &createReq)...)
if resp.Diagnostics.HasError() {
return
}

funcInfo, err := w.Functions.Create(ctx, createReq)
if err != nil {
resp.Diagnostics.AddError("failed to create function", err.Error())
}

resp.Diagnostics.Append(waitForFunction(ctx, w, funcInfo)...)
if resp.Diagnostics.HasError() {
return
}

resp.Diagnostics.Append(converters.GoSdkToTfSdkStruct(ctx, funcInfo, &planFunc)...)
if resp.Diagnostics.HasError() {
return
}

resp.Diagnostics.Append(resp.State.Set(ctx, planFunc)...)
}

func (r *FunctionResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
ctx = pluginfwcontext.SetUserAgentInResourceContext(ctx, resourceName)
w, diags := r.Client.GetWorkspaceClient()
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}

var planFunc catalog_tf.FunctionInfo
resp.Diagnostics.Append(req.Plan.Get(ctx, &planFunc)...)
if resp.Diagnostics.HasError() {
return
}

var updateReq catalog.UpdateFunction

resp.Diagnostics.Append(converters.TfSdkToGoSdkStruct(ctx, planFunc, &updateReq)...)
if resp.Diagnostics.HasError() {
return
}

funcInfo, err := w.Functions.Update(ctx, updateReq)
if err != nil {
resp.Diagnostics.AddError("failed to update function", err.Error())
}

resp.Diagnostics.Append(converters.GoSdkToTfSdkStruct(ctx, funcInfo, &planFunc)...)
if resp.Diagnostics.HasError() {
return
}

resp.Diagnostics.Append(resp.State.Set(ctx, planFunc)...)
}

func (r *FunctionResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
ctx = pluginfwcontext.SetUserAgentInResourceContext(ctx, resourceName)

w, diags := r.Client.GetWorkspaceClient()
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}

var stateFunc catalog_tf.FunctionInfo

resp.Diagnostics.Append(req.State.Get(ctx, &stateFunc)...)
if resp.Diagnostics.HasError() {
return
}

funcName := stateFunc.Name.ValueString()

funcInfo, err := w.Functions.GetByName(ctx, funcName)
if err != nil {
if apierr.IsMissing(err) {
resp.State.RemoveResource(ctx)
return
}
resp.Diagnostics.AddError("failed to get function", err.Error())
return
}

resp.Diagnostics.Append(converters.GoSdkToTfSdkStruct(ctx, funcInfo, &stateFunc)...)
if resp.Diagnostics.HasError() {
return
}

resp.Diagnostics.Append(resp.State.Set(ctx, stateFunc)...)
}

func (r *FunctionResource) Delete(ctx context.Context, req resource.DeleteRequest, resp *resource.DeleteResponse) {
ctx = pluginfwcontext.SetUserAgentInResourceContext(ctx, resourceName)
w, diags := r.Client.GetWorkspaceClient()
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}

var deleteReq catalog_tf.DeleteFunctionRequest
resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root("full_name"), &deleteReq.Name)...)
if resp.Diagnostics.HasError() {
return
}

err := w.Functions.DeleteByName(ctx, deleteReq.Name.ValueString())
if err != nil && !apierr.IsMissing(err) {
resp.Diagnostics.AddError("failed to delete function", err.Error())
}
}
Loading
Loading