Skip to content

Commit

Permalink
feat: add overview on generating bindings
Browse files Browse the repository at this point in the history
  • Loading branch information
@nilslice
nilslice committed Nov 20, 2024
1 parent 71c0263 commit cc5aa3d
Show file tree
Hide file tree
Showing 2 changed files with 175 additions and 40 deletions.
187 changes: 147 additions & 40 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Extism Go PDK

This library can be used to write [Extism Plug-ins](https://extism.org/docs/concepts/plug-in) in Go.
This library can be used to write
[Extism Plug-ins](https://extism.org/docs/concepts/plug-in) in Go.

## Install

Expand All @@ -12,11 +13,17 @@ go get github.com/extism/go-pdk

## Reference Documentation

You can find the reference documentation for this library on [pkg.go.dev](https://pkg.go.dev/github.com/extism/go-pdk).
You can find the reference documentation for this library on
[pkg.go.dev](https://pkg.go.dev/github.com/extism/go-pdk).

## Getting Started

The goal of writing an [Extism plug-in](https://extism.org/docs/concepts/plug-in) is to compile your Go code to a Wasm module with exported functions that the host application can invoke. The first thing you should understand is creating an export. Let's write a simple program that exports a `greet` function which will take a name as a string and return a greeting string. Paste this into your `main.go`:
The goal of writing an
[Extism plug-in](https://extism.org/docs/concepts/plug-in) is to compile your Go
code to a Wasm module with exported functions that the host application can
invoke. The first thing you should understand is creating an export. Let's write
a simple program that exports a `greet` function which will take a name as a
string and return a greeting string. Paste this into your `main.go`:

```go
package main
Expand All @@ -38,38 +45,51 @@ func main() {}

Some things to note about this code:

1. The `//export greet` comment is required. This marks the greet function as an export with the name `greet` that can be called by the host.
1. The `//export greet` comment is required. This marks the greet function as an
export with the name `greet` that can be called by the host.
2. We need a `main` but it is unused.
3. Exports in the Go PDK are coded to the raw ABI. You get parameters from the host by calling [pdk.Input* functions](https://pkg.go.dev/github.com/extism/go-pdk#Input) and you send returns back with the [pdk.Output* functions](https://pkg.go.dev/github.com/extism/go-pdk#Output).
4. An Extism export expects an i32 return code. `0` is success and `1` is a failure.
3. Exports in the Go PDK are coded to the raw ABI. You get parameters from the
host by calling
[pdk.Input* functions](https://pkg.go.dev/github.com/extism/go-pdk#Input) and
you send returns back with the
[pdk.Output* functions](https://pkg.go.dev/github.com/extism/go-pdk#Output).
4. An Extism export expects an i32 return code. `0` is success and `1` is a
failure.

Install the `tinygo` compiler:
Install the `tinygo` compiler:

See https://tinygo.org/getting-started/install/ for instructions for your platform.
See https://tinygo.org/getting-started/install/ for instructions for your
platform.

> Note: while the core Go toolchain has support to target WebAssembly, we find `tinygo` to work well for plug-in code. Please open issues on this repository if you try building with `go build` instead & have problems!
> Note: while the core Go toolchain has support to target WebAssembly, we find
> `tinygo` to work well for plug-in code. Please open issues on this repository
> if you try building with `go build` instead & have problems!
Compile this with the command:

```bash
tinygo build -o plugin.wasm -target wasi main.go
```

We can now test `plugin.wasm` using the [Extism CLI](https://github.com/extism/cli)'s `run`
command:
We can now test `plugin.wasm` using the
[Extism CLI](https://github.com/extism/cli)'s `run` command:

```bash
extism call plugin.wasm greet --input "Benjamin" --wasi
# => Hello, Benjamin!
```

> **Note**: Currently `wasi` must be provided for all Go plug-ins even if they don't need system access, however this will eventually be optional.
> **Note**: Currently `wasi` must be provided for all Go plug-ins even if they
> don't need system access, however this will eventually be optional.
> **Note**: We also have a web-based, plug-in tester called the [Extism Playground](https://playground.extism.org/)
> **Note**: We also have a web-based, plug-in tester called the
> [Extism Playground](https://playground.extism.org/)
### More Exports: Error Handling

Suppose we want to re-write our greeting module to never greet Benjamins. We can use [pdk.SetError](https://pkg.go.dev/github.com/extism/go-pdk#SetError) or [pdk.SetErrorString](https://pkg.go.dev/github.com/extism/go-pdk#SetErrorString):
Suppose we want to re-write our greeting module to never greet Benjamins. We can
use [pdk.SetError](https://pkg.go.dev/github.com/extism/go-pdk#SetError) or
[pdk.SetErrorString](https://pkg.go.dev/github.com/extism/go-pdk#SetErrorString):

```go
//export greet
Expand Down Expand Up @@ -100,8 +120,9 @@ echo $?

### Json

Extism export functions simply take bytes in and bytes out. Those can be whatever you want them to be. A common and simple way to get more complex types to and from the host is with json:

Extism export functions simply take bytes in and bytes out. Those can be
whatever you want them to be. A common and simple way to get more complex types
to and from the host is with json:

```go
type Add struct {
Expand Down Expand Up @@ -141,7 +162,9 @@ extism call plugin.wasm add --input='{"a": 20, "b": 21}' --wasi
## Configs

Configs are key-value pairs that can be passed in by the host when creating a
plug-in. These can be useful to statically configure the plug-in with some data that exists across every function call. Here is a trivial example using [pdk.GetConfig](https://pkg.go.dev/github.com/extism/go-pdk#GetConfig):
plug-in. These can be useful to statically configure the plug-in with some data
that exists across every function call. Here is a trivial example using
[pdk.GetConfig](https://pkg.go.dev/github.com/extism/go-pdk#GetConfig):

```go
//export greet
Expand All @@ -157,7 +180,8 @@ func greet() int32 {
}
```

To test it, the [Extism CLI](https://github.com/extism/cli) has a `--config` option that lets you pass in `key=value` pairs:
To test it, the [Extism CLI](https://github.com/extism/cli) has a `--config`
option that lets you pass in `key=value` pairs:

```bash
extism call plugin.wasm greet --config user=Benjamin
Expand All @@ -168,7 +192,7 @@ extism call plugin.wasm greet --config user=Benjamin

Variables are another key-value mechanism but it's a mutable data store that
will persist across function calls. These variables will persist as long as the
host has loaded and not freed the plug-in.
host has loaded and not freed the plug-in.

```go
//export count
Expand All @@ -181,12 +205,19 @@ func count() int32 {
}
```

> **Note**: Use the untyped variants [pdk.SetVar(string, []byte)](https://pkg.go.dev/github.com/extism/go-pdk#SetVar) and [pdk.GetVar(string) []byte](https://pkg.go.dev/github.com/extism/go-pdk#GetVar) to handle your own types.
> **Note**: Use the untyped variants
> [pdk.SetVar(string, []byte)](https://pkg.go.dev/github.com/extism/go-pdk#SetVar)
> and
> [pdk.GetVar(string) []byte](https://pkg.go.dev/github.com/extism/go-pdk#GetVar)
> to handle your own types.
## Logging

Because Wasm modules by default do not have access to the system, printing to stdout won't work (unless you use WASI).
Extism provides a simple [logging function](https://pkg.go.dev/github.com/extism/go-pdk#Log) that allows you to use the host application to log without having to give the plug-in permission to make syscalls.
Because Wasm modules by default do not have access to the system, printing to
stdout won't work (unless you use WASI). Extism provides a simple
[logging function](https://pkg.go.dev/github.com/extism/go-pdk#Log) that allows
you to use the host application to log without having to give the plug-in
permission to make syscalls.

```go
//export log_stuff
Expand All @@ -210,11 +241,15 @@ extism call plugin.wasm log_stuff --wasi --log-level=debug
2023/10/12 12:11:23 An error log!
```

> *Note*: From the CLI you need to pass a level with `--log-level`. If you are running the plug-in in your own host using one of our SDKs, you need to make sure that you call `set_log_file` to `"stdout"` or some file location.
> _Note_: From the CLI you need to pass a level with `--log-level`. If you are
> running the plug-in in your own host using one of our SDKs, you need to make
> sure that you call `set_log_file` to `"stdout"` or some file location.
## HTTP

Sometimes it is useful to let a plug-in [make HTTP calls](https://pkg.go.dev/github.com/extism/go-pdk#HTTPRequest.Send). [See this example](example/http/tiny_main.go)
Sometimes it is useful to let a plug-in
[make HTTP calls](https://pkg.go.dev/github.com/extism/go-pdk#HTTPRequest.Send).
[See this example](example/http/tiny_main.go)

```go
//export http_get
Expand All @@ -232,7 +267,9 @@ func httpGet() int32 {
}
```

By default, Extism modules cannot make HTTP requests unless you specify which hosts it can connect to. You can use `--alow-host` in the Extism CLI to set this:
By default, Extism modules cannot make HTTP requests unless you specify which
hosts it can connect to. You can use `--alow-host` in the Extism CLI to set
this:

```
extism call plugin.wasm http_get --wasi --allow-host='*.typicode.com'
Expand All @@ -241,24 +278,30 @@ extism call plugin.wasm http_get --wasi --allow-host='*.typicode.com'

## Imports (Host Functions)

Like any other code module, Wasm not only let's you export functions to the outside world, you can
import them too. Host Functions allow a plug-in to import functions defined in the host. For example,
if you host application is written in Python, it can pass a Python function down to your Go plug-in
where you can invoke it.
Like any other code module, Wasm not only let's you export functions to the
outside world, you can import them too. Host Functions allow a plug-in to import
functions defined in the host. For example, if you host application is written
in Python, it can pass a Python function down to your Go plug-in where you can
invoke it.

This topic can get fairly complicated and we have not yet fully abstracted the Wasm knowledge you need
to do this correctly. So we recommend reading our [concept doc on Host Functions](https://extism.org/docs/concepts/host-functions) before you get started.
This topic can get fairly complicated and we have not yet fully abstracted the
Wasm knowledge you need to do this correctly. So we recommend reading our
[concept doc on Host Functions](https://extism.org/docs/concepts/host-functions)
before you get started.

### A Simple Example

Host functions have a similar interface as exports. You just need to declare them as extern on the top of your main.go. You only declare the interface as it is the host's responsibility to provide the implementation:
Host functions have a similar interface as exports. You just need to declare
them as extern on the top of your main.go. You only declare the interface as it
is the host's responsibility to provide the implementation:

```go
//go:wasmimport extism:host/user a_python_func
func aPythonFunc(uint64) uint64
```

We should be able to call this function as a normal Go function. Note that we need to manually handle the pointer casting:
We should be able to call this function as a normal Go function. Note that we
need to manually handle the pointer casting:

```go
//export hello_from_python
Expand All @@ -272,13 +315,14 @@ func helloFromPython() int32 {
pdk.OutputString(response)
return 0
}

```

### Testing it out

We can't really test this from the Extism CLI as something must provide the implementation. So let's
write out the Python side here. Check out the [docs for Host SDKs](https://extism.org/docs/concepts/host-sdk) to implement a host function in a language of your choice.
We can't really test this from the Extism CLI as something must provide the
implementation. So let's write out the Python side here. Check out the
[docs for Host SDKs](https://extism.org/docs/concepts/host-sdk) to implement a
host function in a language of your choice.

```python
from extism import host_fn, Plugin
Expand All @@ -296,7 +340,7 @@ def a_python_func(input: str) -> str:
```

Now when we load the plug-in we pass the host function:

```python
manifest = {"wasm": [{"path": "/path/to/plugin.wasm"}]}
plugin = Plugin(manifest, functions=[a_python_func], wasi=True)
Expand All @@ -312,7 +356,11 @@ python3 app.py

## Reactor modules

Since TinyGo doesn't support [Reactor modules](https://dylibso.com/blog/wasi-command-reactor/) yet, If you want to use WASI inside your Reactor module functions (exported functions other than `main`), you'll need to import `wasi-reactor` module which makes sure libc and go runtime are properly initialized:
Since TinyGo doesn't support
[Reactor modules](https://dylibso.com/blog/wasi-command-reactor/) yet, If you
want to use WASI inside your Reactor module functions (exported functions other
than `main`), you'll need to import `wasi-reactor` module which makes sure libc
and go runtime are properly initialized:

```go
package main
Expand Down Expand Up @@ -348,6 +396,65 @@ extism call ./reactor.wasm read_file --input "./test.txt" --allow-path . --wasi

Note: this is not required if you only have the `main` function.

### Reach Out!
## Generating Bindings

It's often very useful to define a schema to describe the function signatures
and types you want to use between Extism SDK and PDK languages.

[XTP Bindgen](https://github.com/dylibso/xtp-bindgen) is an open source
framework to generate PDK bindings for Extism plug-ins. It's used by the
[XTP Platform](https://www.getxtp.com/), but can be used outside of the platform
to define any Extism compatible plug-in system.

### 1. Install the `xtp` CLI.

See installation instructions
[here](https://docs.xtp.dylibso.com/docs/cli#installation).

### 2. Create a schema using our OpenAPI-inspired IDL:

```yaml
version: v1-draft
exports:
CountVowels:
input:
type: string
contentType: text/plain; charset=utf-8
output:
$ref: "#/components/schemas/VowelReport"
contentType: application/json
# components.schemas defined in example-schema.yaml...
```

> See an example in [example-schema.yaml](./example-schema.yaml), or a full
> "kitchen sink" example on
> [the docs page](https://docs.xtp.dylibso.com/docs/concepts/xtp-schema/).
### 3. Generate bindings to use from your plugins:

```
xtp plugin init --schema-file ./example-schema.yaml
1. TypeScript
> 2. Go
3. Rust
4. Python
5. C#
6. Zig
7. C++
8. GitHub Template
9. Local Template
```

This will create an entire boilerplate plugin project for you to get started
with. Implement the empty function(s), and run `xtp plugin build` to compile
your plugin.

> For more information about XTP Bindgen, see the
> [dylibso/xtp-bindgen](https://github.com/dylibso/xtp-bindgen) repository and
> the official
> [XTP Schema documentation](https://docs.xtp.dylibso.com/docs/concepts/xtp-schema).
## Reach Out!

Have a question or just want to drop in and say hi? [Hop on the Discord](https://extism.org/discord)!
Have a question or just want to drop in and say hi?
[Hop on the Discord](https://extism.org/discord)!
28 changes: 28 additions & 0 deletions example-schema.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# yaml-language-server: $schema=https://xtp.dylibso.com/assets/wasm/schema.json
# Learn more at https://docs.xtp.dylibso.com/docs/concepts/xtp-schema
version: v1-draft
exports:
CountVowels:
input:
type: string
contentType: text/plain; charset=utf-8
output:
$ref: "#/components/schemas/VowelReport"
contentType: application/json
components:
schemas:
VowelReport:
description: The result of counting vowels on the Vowels input.
properties:
count:
type: integer
format: int32
description: The count of vowels for input string.
total:
type: integer
format: int32
description: The cumulative amount of vowels counted, if this keeps state across multiple function calls.
nullable: true
vowels:
type: string
description: The set of vowels used to get the count, e.g. "aAeEiIoOuU"

0 comments on commit cc5aa3d

Please sign in to comment.