elixir-avro

elixir-avro is a library designed to facilitate the conversion of Avro schemas into Elixir code. This library bridges the gap between Avro schema definitions and Elixir code, enabling seamless integration of Avro-defined data structures into Elixir projects.

Installation

If available in Hex, the package can be installed by adding elixir_avro to your list of dependencies in mix.exs:

def deps do
  [
    {:elixir_avro, "~> 0.1.0"}
  ]
end

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/elixir_avro.

Usage

Run this command to generate Elixir code from Avro schema files:

This will generate Elixir code for the Avro schema files in the avro directory, one for each Avro type defined in the schemas.

Example

Given the following Avro schema definition in avro/user.avsc:

{
  "name": "User",
  "namespace": "registry",
  "fullname": "registry.User",
  "doc": "User registry information",
  "type": "record",
  "fields": [
    {
      "name": "firstname",
      "type": "string"
    },
    {
      "name": "lastname",
      "type": "string"
    },
    {
      "name": "age",
      "type": "int"
    },
    {
      "name": "role",
      "type": {
        "type": "enum",
        "name": "Role",
        "doc": "User's role.",
        "fullname": "registry.user.Role",
        "namespace": "registry.user",
        "symbols": [
          "ADMIN",
          "USER"
        ]
      }
    }
  ]
}

Running the following command:

mix elixir_avro_codegen --schemas-path avro/ --target-path lib --prefix MyApp.Avro

Will generate the following Elixir code in lib/:

# lib/my_app/avro/registry/user.ex
defmodule MyApp.Avro.Registry.User do
  @moduledoc """
  DO NOT EDIT MANUALLY: This module was automatically generated from an AVRO schema.

  ### Description
  User registry information

  ### Fields
  - __fullname__: User's full name
  - __role__: User's role.
  """

  use TypedStruct

  alias ElixirAvro.AvroType.Value.Decoder
  alias ElixirAvro.AvroType.Value.Encoder

  @expected_keys MapSet.new(["fullname", "role"])

  typedstruct do
    field :fullname, String.t(), enforce: true
    field :role, MyApp.Avro.Registry.User.Role.t(), enforce: true
  end

  @module_prefix MyApp.Avro

  def to_avro(%__MODULE__{} = struct) do
    {:ok,
     %{
      # ...
     }}
  end

  def from_avro(%{"fullname" => fullname, "role" => role}) do
    {:ok,
     %__MODULE__{
      # ...
     }}
  rescue
    e -> {:error, inspect(e)}
  end

  # ...
end

# lib/my_app/avro/registry/user/role.ex
defmodule MyApp.Avro.Registry.User.Role do
  @moduledoc """
  DO NOT EDIT MANUALLY: This module was automatically generated from an AVRO schema.

  ### Description
  User's role.

  """

  use ElixirAvro.Macro.ElixirEnum

  @values ["ADMIN", "USER"]

  def to_avro(value) do
    {:ok, to_avro_string(value)}
  rescue
    _ -> {:error, :invalid_enum_value}
  end

  def from_avro(value) do
    {:ok, from_avro_string(value)}
  rescue
    _ -> {:error, :invalid_enum_string_value}
  end
end

The generated path is a concatenation of the --prefix option (snake-cased) and the Avro schema type's namespace.

CLI Options

Usage: mix elixir_avro_codegen <-s directory> <-t directory> <-p name> [-v]
-s, --schemas-path <directory>    The path to the directory containing the Avro schema files.
-t, --target-path <directory>     The path to the directory where the generated Elixir code will be saved.
-p, --prefix <name>               The prefix to be used for the generated Elixir modules.
-v, --verbose                     Enable verbose output.

Compile-time generation

In your mix.exs file, in project function, add the following configuration:

  def project do
    [
      # ...
      elixir_avro_codegen: [
        schema_path: "avro",
        target_path: "avro",
        prefix: "MyApp.Avro"
      ],
      compilers: Mix.compilers() ++ [:elixir_avro_codegen],
      # ...
    ]
  end

The parameters used in the elixir_avro_codegen configuration are the same as the CLI options.

Custom logical types

To enable custom logical types support, you can create a module that conforms to the ElixirAvro.AvroType.LogicalType behaviour. Implementing this behaviour necessitates the user to define encode and decode functions. Here's an example:

defmodule My.Custom.LogicalType.Enum do
  @moduledoc false
  @behaviour ElixirAvro.AvroType.LogicalType
  @type t :: :admin | :user
  @values ["admin", "user"]
  defstruct [:value]

  @impl ElixirAvro.AvroType.LogicalType
  def decode(value) when value in @values, do: {:ok, %__MODULE__{value: String.to_atom(value)}}
  def decode(value), do: {:error, "Unknown value #{value}"}

  @impl ElixirAvro.AvroType.LogicalType
  def encode(%__MODULE__{value: value}), do: {:ok, Atom.to_string(value)}
end

Then, in order to let the library be able to use the custom logical type, you need to add the following configuration in your config file:

config :elixir_avro, :custom_logical_types, %{
  {"string", "custom-logical-enum"} => My.Custom.LogicalType.Enum
}

The key, which consists of a tuple containing the primitive type, will be utilized for deserializing the Avro custom logical type, along with its corresponding name as specified in the Avro file.

License

The MIT License (MIT)

Copyright (c) 2020 Prima.it

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.