Basic Buf Example

.gitignore

@@ -1,2 +1,32 @@
+.vscode/
+.bloop/
+.metals/
+project/.bloop/
+metals.sbt
+
 *.class
 *.log
+*.swp
+
+# sbt specific
+.bsp
+.cache/
+.history/
+.lib/
+dist/*
+target/
+lib_managed/
+src_managed/
+project/boot/
+project/plugins/project/
+
+# Scala-IDE specific
+.scala_dependencies
+.worksheet
+.idea
+
+
+#Probably not the best idea normally but to keep this repo clean and let people run stuff themselves
+/src/main/scala/com/example/petstore/generated
+/src/main/scala/io
+/src/main/scala/scalapb

.scalafix.conf

@@ -0,0 +1,17 @@
+rules = [OrganizeImports]
+
+OrganizeImports {
+  groupedImports = Merge
+  importsOrder = SymbolsFirst
+  importSelectorsOrder = SymbolsFirst
+  groups = [
+    "com.example.",
+    "com.",
+    "zio.",
+    "org.",
+    "scala.",
+    "java.",
+    "javax.",
+    "*"
+  ]
+}

.scalafmt.conf

@@ -0,0 +1,5 @@
+version = "3.7.2"
+rewrite.rules = [SortImports, RedundantBraces]
+maxColumn = 120
+align.preset = most
+runner.dialect = scala213source3

README.md

@@ -0,0 +1,175 @@
+# ScalaPb Buf Examples.
+
+This repository contains examples of [Buf](https://buf.build/) and it's use with Scala.
+
+## What is Buf?
+
+Buf is a command-line interface tool that provides advanced functionality for working with Protocol Buffers, a popular binary serialization format. It offers features such as linting, compatibility checking, code generation, and dependency management, which can streamline the development workflow for Protocol Buffers-based projects. Buf configuration is defined through YAML files.
+
+## Prerequisites
+
+This guide assumes that you have SBT installed if you wan to compile and run the example code.
+
+## Getting Started
+
+To install Buf, follow these steps:
+
+1. Download the appropriate version of Buf for your operating system from the [official website](https://docs.buf.build/installation).
+2. Choose your operating system.
+3. Follow the instructions on the website.
+
+
+## Usage
+
+Once Buf is installed, you can use it to validate, lint, and generate code from your Protocol Buffer files. Here are a few basic commands to get you started:
+
+### Lint your project
+
+The example project uses a datamodel object (`User`) as a request and response type, which is suboptimal. It is recommended to use separate request and response objects in Protocol Buffers for RPC as it leads to better encapsulation, flexibility, and type safety. This approach helps encapsulate the data being sent and received, allows for easier modifications without breaking backwards compatibility, and ensures errors are caught at compile time.
+
+Running the following command will produce some errors:
+
+```
+buf lint
+```
+
+This will produce the following output:
+
+```
+src/main/protobuf/petstore/v1/petstore.proto:44:5:"petstore.v1.User" is used as the request or response type for multiple RPCs.
+src/main/protobuf/petstore/v1/petstore.proto:44:54:RPC response type "User" should be named "ListUsersResponse" or "PetStoreServiceListUsersResponse".
+src/main/protobuf/petstore/v1/petstore.proto:45:5:"petstore.v1.User" is used as the request or response type for multiple RPCs.
+src/main/protobuf/petstore/v1/petstore.proto:45:28:RPC request type "User" should be named "StoreUsersRequest" or "PetStoreServiceStoreUsersRequest".
+src/main/protobuf/petstore/v1/petstore.proto:46:5:"petstore.v1.User" is used as the request or response type for multiple RPCs.
+src/main/protobuf/petstore/v1/petstore.proto:46:5:RPC "BulkUsers" has the same type "petstore.v1.User" for the request and response.
+src/main/protobuf/petstore/v1/petstore.proto:46:27:RPC request type "User" should be named "BulkUsersRequest" or "PetStoreServiceBulkUsersRequest".
+src/main/protobuf/petstore/v1/petstore.proto:46:49:RPC response type "User" should be named "BulkUsersResponse" or "PetStoreServiceBulkUsersResponse".
+```
+
+Note that Buf suggests consistent naming, which also applies to other request/response objects. A consistent naming scheme improves code readability and is easier to maintain.
+
+If you don't like the default configuration, you can alter it to suit your preferences. See the [documentation](https://buf.build/docs/breaking/usage/#key-concepts) for configuration details.
+
+## Format your project
+No one likes messy code. That's why Buf comes with a handy feature to format your proto files and make them neat and tidy. Use the following command to format this project:
+
+```
+buf format -w
+```
+The `-w` option will write in place.
+
+To see the difference in formatting run:
+```
+git diff
+```
+
+Your peers will thank you for it!
+
+### Detect Breaking Changes
+Let's shake things up by introducing a breaking change to our protocol buffer definition. To do this, we'll delete the id field from the User message. Once you've made the change, run the following command:
+```
+buf breaking --against '.git#branch=master'
+```
+This will check for any breaking changes in the codebase and report them. In our case, you should see an error that reads:
+```
+src/main/protobuf/petstore/v1/petstore.proto:11:1:Previously present field "1" with name "id" on message "User" was deleted.
+```
+
+Buf provides a powerful compatibility checking tool that can also catch other types of errors such as type changes. You can configure different levels of compatibility checks depending on your needs. More information and configuration options can be found in the breaking [breaking documentation](https://buf.build/docs/breaking/overview/#key-concepts).
+
+It's worth noting that Buf doesn't currently check for breaking changes with extensions made by the `protoc-validate` plugin.
+
+### Buf Schema Registry (BSR)
+Buf Schema Registry (BSR) is the place to be for storing your Protocol Buffers schemas. It provides a centralized location for managing your schema files, allowing you to easily store, share, and version them. With BSR, there's no need to copy and paste your protos or publish them as jars - everything is taken care of for you!
+
+With BSR, you can ensure consistency and compatibility across services, and simplify the process of integrating with external services. You can follow this [link](https://buf.build/explore)  to explore protobufs published to BSR. Published modules also include viewable documentation of the types published, making it even easier to integrate.
+
+To publish your protos, follow the well-documented steps outlined in [Buf's documentation](https://buf.build/docs/bsr/quick-start/#step-1:-sign-up-for-a-buf-account).
+
+To pull a dependency, simply add it to your `buf.yaml` file. For instance, if you want to use googleapis, add the following:
+```
+deps:
+  - buf.build/googleapis/googleapis:<GIT_COMMIT_TAG>
+```
+Here, <GIT_COMMIT_TAG> is the version you want to use. If you don't include it, Buf will always import the latest version. Once you've added the dependency, execute the following command to begin the import process:
+```
+buf mod update
+```
+And that's it! Buf will resolve the types of the dependencies and understand their use in your local environment. Additionally, the protobuf types used will also be generated. And if your protobuf files aren't intended for public consumption, Buf also supports private packages.
+
+An example can be found in the `buf.yaml` file found in the petstore directory.
+
+In this example we are using the validate protos used by scalapb-validate:
+```
+deps:
+  - buf.build/envoyproxy/protoc-gen-validate:728c81676f9e54d3571603b90b34c0e6419770c6
+```
+The other two dependencies are necessary for a demo of local plugins, they can be ignored for now. If you decide that a dependency is no longer needed you can remove it and run:
+
+```
+buf mod prune
+```
+
+By adding this dependency we get access to `import "validate/validate.proto";` and the proto extensions available to it. However this is not limited to only extensions, this is also a great way to share common dependencies.
+
+### Code Generation
+Oh no! Looks like this project is missing all the Scala types that are needed to compile and run the server. But with Buf we can easily fix this. So let's generate the code we need by running this command:
+```
+buf generate
+```
+
+After running this, you should see all the Scala code and be able to compile and run the server. The code generation is performed by reading the `buf.gen.yaml` directory and executing the remote plugins listed there. Buf supports a range of plugins, which can be found on the [plugins page](https://buf.build/plugins).
+
+Plugins for Scala specifically that currently exist are
+- [Base ScalaPb Compiler](https://buf.build/community/scalapb-scala)
+- [ZIO GRPC](https://buf.build/community/scalapb-zio-grpc)
+
+**There are a list of plugins, however, that can be added quite easily, but due to the maintenance cost by the Buf team, they have not yet been merged. Please drop a comment or +1 in the linked issues below if you want to see them! Bonus points if you feel like your organization would make use of them.**
+
+Two popular GRPC implementations that can be merged fairly quickly are [FS2 GRPC](https://github.com/bufbuild/plugins/issues/305) and [Akka GRPC](https://github.com/bufbuild/plugins/issues/306). [Scalapb Validate](https://github.com/bufbuild/plugins/issues/304) is also missing, but would likely require more work than the other teams as Buf's test bench for added plugins does not support chained plugins.
+
+Note that plugins do not take care of importing the runtime, so you will need to manually align the runtime dependency version with the plugin version you have defined. In case of the ZIO example, you can refer to the build.sbt file for the dependencies needed. For other runtimes, you will need to include equivalent dependencies.
+
+### How to include a custom protoc plugin in Buf
+When using a custom protoc plugin that is not available remotely, it is not as straightforward as adding a remote plugin. This is especially true if you want to ensure that your team's plugin version is kept in sync. Here are the general steps to follow (with full documentation and examples [here](https://buf.build/docs/generate/usage/#2.-define-a-module)):
+
+1. Download your protoc plugin
+2. Install the protoc plugin on your $PATH using protoc-gen-<PLUGIN_NAME>
+3. In your buf.gen.yaml file, include your plugin in the list of plugins using the plugin: <PLUGIN_NAME> syntax.
+
+Keep in mind that this approach requires more manual setup and maintenance compared to using a remote plugin. Additionally you may wish to include the version as a suffix to the <PLUGIN_NAME> to ensure consistent behavior across computers, or allow different projects to work with different versions of the same plugin.
+
+
+#### Running the Validate Plugin locally
+
+This example repository uses the validate plugin, and can be used for code generation. To execute the plugin locally follow these steps, note that some commands may require `sudo` or the equivalent in Windows:
+
+[Download scalapb-validate-0.3.4](https://repo1.maven.org/maven2/com/thesamet/scalapb/protoc-gen-scalapb-validate/0.3.4/protoc-gen-scalapb-validate-0.3.4-unix.sh)
+
+Once downloaded move the file onto your path. For example:
+```
+mv protoc-gen-scalapb-validate-0.3.4-unix.sh /usr/local/bin/protoc-gen-scalapb-validate-0.3.4
+```
+
+**It is really important that it starts with protoc-gen-* as buf assumes the plugins are named with this prefix.**
+
+Make sure that the script is executable
+```
+chown +x /usr/local/bin/protoc-gen-scalapb-validate-0.3.4
+```
+And that's it! Buf can now execute your local plugin!
+
+
+To see this in action, uncomment out the sections as labeled in the `petstore.proto` file and the `buf.gen.yaml` file and the `build.sbt` file.
+
+Here is why:
+* `petstore.proto` - the commented section shows how to configure some of the validation features, it also changes the code to do something meanigfully different (validate_at_construction) when you run the server
+* `buf.gen.yaml` - this is the part that actually instructs buf to use the local installed plugin
+* `build.sbt` - the validation plugin refers to some types that are not generated, not all plugins will need this, but like in the grpc example we need the runtime to work.
+
+Run `buf generate --include-imports` to generate the new code. Now, if you run the example server, you will now see that the fields marked for validation are now validated prior to the execution of the endpoint.
+
+
+## Conclusion
+
+That's all to this guide however this guide only gives only a high level overview of what can be achieved and configured with Buf. Buf's resources and documentation are comprehensive, so please review that for more details. If there is something this guide missed, or can be made more clear, please feel free to drop a PR.

buf.gen.yaml

@@ -0,0 +1,14 @@
+version: v1
+plugins:
+  - plugin: buf.build/community/scalapb-scala:v0.11.13
+    out: src/main/scala
+    opt: 
+     - grpc
+  #  Uncomment this part once you have scalapb-validate installed locally
+  # - plugin: scalapb-validate-0.3.4
+  #   out: src/main/scala
+  #   opt:
+  #    - grpc
+  - plugin: buf.build/community/scalapb-zio-grpc:v0.5.3
+    out: src/main/scala
+

buf.work.yaml

@@ -0,0 +1,3 @@
+version: v1
+directories:
+  - src/main/protobuf

build.sbt

@@ -0,0 +1,20 @@
+
+scalaVersion := "2.13.8"
+
+name := "scalapb-buf-examples"
+organization := "scalapb"
+version := "1.0"
+
+ThisBuild / scalafixDependencies += "com.github.liancheng" %% "organize-imports" % "0.6.0"
+scalacOptions += "-Wunused"
+
+libraryDependencies ++= 
+    Seq(
+        "com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion,
+        "com.thesamet.scalapb" %% "scalapb-runtime-grpc" % scalapb.compiler.Version.scalapbVersion,
+        "io.grpc" % "grpc-netty-shaded" % "1.54.0",
+        "com.thesamet.scalapb.zio-grpc" %% "zio-grpc-core" % "0.5.3",
+        // Uncomment the following line for the local valdiation plugin, this is needed
+        // to get some of the types that the code generator references.
+        // "com.thesamet.scalapb" %% "scalapb-validate-core" % "0.3.4",
+    )

project/build.properties

@@ -0,0 +1 @@
+sbt.version=1.8.2

project/plugins.sbt

@@ -0,0 +1,4 @@
+addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.0")
+addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.5.0")
+addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.10.4")
+libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.11.11"

src/main/protobuf/buf.lock

@@ -0,0 +1,15 @@
+# Generated by buf. DO NOT EDIT.
+version: v1
+deps:
+  - remote: buf.build
+    owner: envoyproxy
+    repository: protoc-gen-validate
+    commit: 45685e052c7e406b9fbd441fc7a568a5
+  - remote: buf.build
+    owner: scalapb
+    repository: scalapb
+    commit: 4b8e0968b1ea452983e157014defba1a
+  - remote: buf.build
+    owner: scalapb
+    repository: scalapb-validate
+    commit: 0e9f937482594692842b553afa3e1b21

src/main/protobuf/buf.yaml

@@ -0,0 +1,11 @@
+version: v1
+deps:
+  - buf.build/scalapb/scalapb-validate:v0.3.4
+  - buf.build/envoyproxy/protoc-gen-validate:728c81676f9e54d3571603b90b34c0e6419770c6
+  - buf.build/scalapb/scalapb:v0.11.11
+breaking:
+  use:
+    - FILE
+lint:
+  use:
+    - DEFAULT

src/main/protobuf/petstore/v1/petstore.proto

@@ -0,0 +1,62 @@
+syntax = "proto3";
+package petstore.v1;
+
+option java_package = "com.example.petstore.generated";
+
+import "validate/validate.proto";
+
+// Uncomment this block out after you have installed validate protoc plugin to do this part of the demo
+// import "scalapb/scalapb.proto";
+// import "scalapb/validate/validate.proto"; //TODO: Current publish puts it in an extra dir, should be scalapb/validate.proto
+
+// option (scalapb.options) = {
+//   scope: FILE
+//   [scalapb.validate.file] {
+//       validate_at_construction: true
+//       insert_validator_instance: true
+//       skip: false
+//   }
+// };
+
+message Pet {
+  int32 id = 1;
+  string name = 2;
+}
+
+message User {
+  int32 id = 1;
+  string username = 2;
+  string email = 3;
+  string phone = 4;
+}
+
+message PetByIdRequest {
+  int32 id = 1;
+}
+
+message UserByNameRequest {
+  string username = 1 [(validate.rules).string.min_len = 3];
+}
+
+message PetByIdResponse {
+  Pet pet = 1 [(validate.rules).message.required = true];
+}
+
+message UserByNameResponse {
+  User user = 1;
+}
+
+message ListUsersRequest {
+  string msg = 1;
+}
+message StoreUsersResponse {
+  string msg = 1;
+}
+
+service PetStoreService {
+  rpc PetById(PetByIdRequest) returns (PetByIdResponse);
+  rpc UserByName(UserByNameRequest) returns (UserByNameResponse);
+  rpc ListUsers(ListUsersRequest) returns (stream User);
+  rpc StoreUsers(stream User) returns (StoreUsersResponse);
+  rpc BulkUsers(stream User) returns (stream User);
+}

src/main/scala/com/example/petstore/impl/PetStoreServer.scala

@@ -0,0 +1,20 @@
+package com.example.petstore.impl
+
+import com.example.petstore.generated.petstore.User
+
+import zio.{RefM, ZEnv}
+
+import scalapb.zio_grpc.{ServerMain, ServiceList}
+
+object PetStoreServer extends ServerMain {
+  override def port: Int = 8080
+
+  val createPetStore =
+    for {
+      userState <- RefM.make(Map.empty[String, User])
+    } yield new ZioPetstoreImpl(userState)
+
+  override def services: ServiceList[ZEnv] =
+    ServiceList.addM(createPetStore)
+
+}