-
Notifications
You must be signed in to change notification settings - Fork 44
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[ISSUE #148] Add connectors documents
- Loading branch information
Showing
8 changed files
with
298 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # Open Connect | ||
|
|
||
| ## Connector | ||
| A connector is an image or instance that interacts with a specific external service or underlying data source (e.g., Databases) on behalf of user applications. A connector is either a Source or a Sink. | ||
|
|
||
| ## Source | ||
|
|
||
| A source connector obtains data from an underlying data producer, and delivers it to targets after original data has been transformed into CloudEvents. It doesn't limit the way how a source retrieves data. (e.g., A source may pull data from a message queue or act as an HTTP server waiting for data sent to it). | ||
|
|
||
| ## Sink | ||
|
|
||
| A sink connector receives CloudEvents and does some specific business logics. (e.g., A MySQL Sink extracts useful data from CloudEvents and writes them to a MySQL database). | ||
|
|
||
| ## CloudEvents | ||
| A specification for describing event data in common formats to provide interoperability across services, platforms and systems. | ||
|
|
||
| ## Implements | ||
|
|
||
| Add a new connector by implementing the source/sink interface using [eventmesh-openconnect-java](https://github.com/apache/eventmesh/tree/master/eventmesh-openconnect/eventmesh-openconnect-java). | ||
|
|
||
| ## Technical Solution | ||
|
|
||
| ### Structure and process | ||
|  | ||
|
|
||
| ### Design Detail | ||
|  | ||
|
|
||
| ### Describe | ||
| **Worker** | ||
|
|
||
| Worker is divided into Source Worker and Sink Worker, which are triggered by the `Application` class and implement the methods of the `ConnectorWorker` interface respectively, which include the worker's running life cycle, and the worker carries the running of the connector. Workers can be lightweight and independent through mirroring Running, the eventmesh-sdk-java module is integrated internally, and the cloudevents protocol is used to interact with eventmesh. Currently, the tcp client is used by default. In the future, support for dynamic configuration can be considered | ||
|
|
||
| **Connector** | ||
|
|
||
| Connectors are divided into Source Connector and Sink Connector. Connectors have their own configuration files and run independently. Workers perform reflective loading and configuration analysis to complete Connector initialization and subsequent operation. Source Connector implements the poll method, and Sink Connector implements The put method uniformly uses `ConnectorRecord` to carry data. Both Source Connector and Sink Connector can operate independently. | ||
|
|
||
| **ConnectorRecord with CloudEvents** | ||
|
|
||
| `ConnectorRecord` is a connector layer data protocol. When workers interact with eventmesh, a protocol adapter needs to be developed to convert `ConnectorRecord` to CloudEvents protocol. | ||
|
|
||
| **Registry** | ||
|
|
||
| The Registry module is responsible for storing the synchronization progress of synchronizing data of different Connector instances, ensuring high availability between multiple Connector images or instances. | ||
|
|
||
| ## Connector Status | ||
|
|
||
| | Connector Name | Source | Sink | | ||
| |:------------------------------------------------:|:-----------:|:-------:| | ||
| | [RocketMQ](eventmesh-connector-rocketmq) | ✅ | ✅ | | ||
| | ChatGPT | ⬜ | ⬜ | | ||
| | ClickHouse | ⬜ | ⬜ | | ||
| | [DingTalk](eventmesh-connector-dingtalk) | ⬜ | ✅ | | ||
| | Email | ⬜ | ⬜ | | ||
| | [Feishu/Lark](eventmesh-connector-lark) | ⬜ | ✅ | | ||
| | [File](eventmesh-connector-file) | ✅ | ✅ | | ||
| | GitHub | ⬜ | ⬜ | | ||
| | [HTTP](eventmesh-connector-http) | ✅ | ⬜ | | ||
| | [Jdbc](eventmesh-connector-jdbc) | ⬜ | ✅ | | ||
| | [Kafka](eventmesh-connector-kafka) | ✅ | ✅ | | ||
| | [Knative](eventmesh-connector-knative) | ✅ | ✅ | | ||
| | [MongoDB](eventmesh-connector-mongodb) | ✅ | ✅ | | ||
| | [OpenFunction](eventmesh-connector-openfunction) | ✅ | ✅ | | ||
| | [Pravega](eventmesh-connector-pravega) | ✅ | ✅ | | ||
| | [Prometheus](eventmesh-connector-prometheus) | ✅ | ⬜ | | ||
| | [Pulsar](eventmesh-connector-pulsar) | ✅ | ✅ | | ||
| | [RabbitMQ](eventmesh-connector-rabbitmq) | ✅ | ✅ | | ||
| | [Redis](eventmesh-connector-redis) | ✅ | ✅ | | ||
| | [S3 File](eventmesh-connector-s3) | ⬜ | ✅ | | ||
| | [Slack](eventmesh-connector-slack) | ⬜ | ✅ | | ||
| | [Spring](eventmesh-connector-spring) | ✅ | ✅ | | ||
| | [WeCom](eventmesh-connector-wecom) | ⬜ | ✅ | | ||
| | [WeChat](eventmesh-connector-wechat) | ⬜ | ✅ | | ||
| | More connectors will be added... | N/A | N/A | |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # Lark Connector | ||
|
|
||
| ## Lark Sink Server Config And Start | ||
|
|
||
| Before using eventmesh-connector-lark to sink events, you need to configure the server. | ||
| - Please customize `sinkEnable`=`true`/`false` in `/resource/server-config.yml` to turn on/off the sink function. | ||
| - Regarding `/resource/sink-config.yml`, only the configuration under `sinkConnectorConfig` is explained here: | ||
| - `connectorName`, specify the connector name | ||
| - (required) `appId`, the appId obtained from lark | ||
| - (required) `appSecret`, the appSecret obtained from lark | ||
| - `receiveIdType`, the type of receiving Id, the default and recommended use is `open_id`. Optional open_id/user_id/union_id/email/chat_id. | ||
| - (Required) `receiveId`, receive Id, needs to correspond to `receiveIdType`. | ||
| - `sinkAsync`, whether to asynchronously sink events | ||
| - `maxRetryTimes`, the maximum number of retransmissions when the sink event fails. The default is 3 times. | ||
| - `retryDelayInMills`, when the sink event fails, the time interval for retransmitting the event. Default is 1s, unit is milliseconds. | ||
|
|
||
|
|
||
| ## Sink CloudEvent To Lark | ||
|
|
||
| When using the eventmesh-connector-lark sinking event, you need to add the corresponding extension filed in CloudEvent: | ||
| - When key=`templatetype4lark`, value=`text`/`markdown`, indicating the text type of the event | ||
| - When the text type is markdown, you can add extension: key=`markdownmessagetitle4lark`, value indicates the title of the event. | ||
| - When key=`atusers4lark`, value=`id-0,name-0;id-1,name-1`, indicating that the event requires `@`certain users | ||
| - It is recommended to use **open_id** for id. | ||
| - When the text is of text type, the id can be **open_id/union_id/user_id**; when the text is of markdown type, the id can be **open_id/user_id**. In particular, when the application type is [custom robot](https://open.larksuite.com/document/ukTMukTMukTM/ucTM5YjL3ETO24yNxkjN) and the text is of markdown type, only the use of **open_id** to `@` the user is supported. | ||
| - When the text is of text type and the id is invalid, name will be used instead for display; when the text is of markdown type and the id is invalid, an exception will be thrown directly (you should try to ensure the correctness of the id, and name can be considered omitted). | ||
| - When key=`atall4lark`, value=`true`/`false`, indicating that the event requires `@` everyone. | ||
|
|
||
|
|
||
| ## Lark Open Platform API | ||
|
|
||
| For the Lark open platform API involved in this module, please click the following link: | ||
| - **Send Message**, please [view here](https://open.larksuite.com/document/server-docs/im-v1/message/create?appId=cli_a5e1bc31507ed00c) | ||
| - **text**, please [view here](https://open.larksuite.com/document/server-docs/im-v1/message-content-description/create_json#c9e08671) | ||
| - **markdown**, please [view here](https://open.larksuite.com/document/common-capabilities/message-card/message-cards-content/using-markdown-tags) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # RabbitMQ Connector | ||
|
|
||
| ## RabbitMQSinkConnector: from eventmesh to rabbitmq. | ||
|
|
||
| 1. launch your rabbitmq server and eventmesh-runtime. | ||
| 2. enable sinkConnector and check `sink-config.yml`. | ||
| 3. send a message to eventmesh with the topic defined in `pubSubConfig.subject` | ||
| ```yaml | ||
| pubSubConfig: | ||
| # default port is 10000 | ||
| meshAddress: your.eventmesh.server:10000 | ||
| subject: TopicTest | ||
| idc: FT | ||
| env: PRD | ||
| group: rabbitmqSink | ||
| appId: 5031 | ||
| userName: rabbitmqSinkUser | ||
| passWord: rabbitmqPassWord | ||
| connectorConfig: | ||
| connectorName: rabbitmqSink | ||
| host: your.rabbitmq.server | ||
| port: 5672 | ||
| username: coyrqpyz | ||
| passwd: passwd | ||
| virtualHost: coyrqpyz | ||
| exchangeType: TOPIC | ||
| # build-in exchangeName or name a new one after you create it in rabbitmq server. | ||
| exchangeName: amq.topic | ||
| # rabbitmq server will create the routingKey and queueName automatically after you connect to it if they aren't exist before. | ||
| routingKey: eventmesh | ||
| queueName: eventmesh | ||
| autoAck: true | ||
| ``` | ||
| ## RabbitMQSourceConnector: from rabbitmq to eventmesh. | ||
| 1. launch your rabbitmq server and eventmesh-runtime. | ||
| 2. enable sourceConnector and check `source-config.yml` (Basically the same as `sink-config.yml`) | ||
| 3. start your `RabbitMQConnectorServer` and you will find the channel in rabbitmq server. | ||
| 4. send a cloudevent message to the queue and then you will receive the message in eventmesh. |
74 changes: 74 additions & 0 deletions
74
...aurus-plugin-content-docs/current/design-document/03-connect/01-open-connect.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # 连接器简介 | ||
|
|
||
| ## 连接器类型 | ||
|
|
||
| 连接器是代表用户应用程序与特定外部服务或底层数据源（例如数据库）交互的镜像或实例。连接器的类型可以是源（Source）或汇（Sink）。 | ||
|
|
||
| ## 数据源（Source 端） | ||
|
|
||
| 源连接器从底层数据生产者获取数据，并在原始数据被转换为 CloudEvents 后将其传递给目标。源连接器不限制源如何检索数据（例如，源可以从消息队列中获取数据，也可以充当等待接收数据的 HTTP 服务器）。 | ||
|
|
||
| ## 数据汇（Sink 端） | ||
|
|
||
| 汇连接器接收 CloudEvents 并执行特定的业务逻辑（例如，MySQL 的汇连接器从 CloudEvents 中提取有用的数据，并将其写入 MySQL 数据库）。 | ||
|
|
||
| ## CloudEvents | ||
| CloudEvents 是一种以通用格式描述事件数据的规范，以提供服务、平台和系统之间的互操作性。 | ||
|
|
||
| ## 实现连接器 | ||
|
|
||
| 使用 [eventmesh-openconnect-java](https://github.com/apache/eventmesh/tree/master/eventmesh-openconnect/eventmesh-openconnect-java) 实现 Source/Sink 接口即可添加新的连接器。 | ||
|
|
||
| ## 技术方案 | ||
| ### 结构与处理流程 | ||
|  | ||
|
|
||
| ### 详细设计 | ||
|  | ||
|
|
||
| ### 描述 | ||
| **Worker** | ||
|
|
||
| Worker分为Source Worker与Sink Worker，由`Application`类进行触发运行，分别实现了`ConnectorWorker`接口的方法，其中包含了worker的运行生命周期，worker承载了connector的运行。Worker可以通过镜像的方式轻量的独立运行，内部集成了eventmesh-sdk-java模块，采用cloudevents协议与eventmesh进行交互，目前默认采用tcp客户端，后续可以考虑支持动态可配 | ||
|
|
||
| **Connector** | ||
|
|
||
| Connector分为Source Connector与Sink Connector，connector有各自的配置文件，以及独立运行的方式，通过worker进行反射加载与配置解析，完成Connector的初始化以及后续运行工作，其中Source Connector实现poll方法，Sink Connector实现put方法，统一使用`ConnectorRecord`承载数据。Source Connector与Sink Connector均可独立运行。 | ||
|
|
||
| **ConnectorRecord with CloudEvents** | ||
|
|
||
| `ConnectorRecord`为connector层数据协议，当worker与eventmesh进行交互时需开发协议适配器进行`ConnectorRecord`到CloudEvents的协议转换。 | ||
|
|
||
| **Registry** | ||
|
|
||
| `Registry`模块负责存储同步不同Connector实例的数据的同步进度，确保多个Connector镜像或实例之间的高可用。 | ||
|
|
||
| ## 连接器实现状态 | ||
|
|
||
| | 连接器名称 | 源 | 汇 | | ||
| |:------------------------------------------------:|:---:|:---:| | ||
| | [RocketMQ](eventmesh-connector-rocketmq) | ✅ | ✅ | | ||
| | ChatGPT | ⬜ | ⬜ | | ||
| | ClickHouse | ⬜ | ⬜ | | ||
| | [DingTalk](eventmesh-connector-dingtalk) | ⬜ | ✅ | | ||
| | Email | ⬜ | ⬜ | | ||
| | [Feishu/Lark](eventmesh-connector-lark) | ⬜ | ✅ | | ||
| | [File](eventmesh-connector-file) | ✅ | ✅ | | ||
| | GitHub | ⬜ | ⬜ | | ||
| | [HTTP](eventmesh-connector-http) | ✅ | ⬜ | | ||
| | [Jdbc](eventmesh-connector-jdbc) | ⬜ | ✅ | | ||
| | [Kafka](eventmesh-connector-kafka) | ✅ | ✅ | | ||
| | [Knative](eventmesh-connector-knative) | ✅ | ✅ | | ||
| | [MongoDB](eventmesh-connector-mongodb) | ✅ | ✅ | | ||
| | [OpenFunction](eventmesh-connector-openfunction) | ✅ | ✅ | | ||
| | [Pravega](eventmesh-connector-pravega) | ✅ | ✅ | | ||
| | [Prometheus](eventmesh-connector-prometheus) | ✅ | ⬜ | | ||
| | [Pulsar](eventmesh-connector-pulsar) | ✅ | ✅ | | ||
| | [RabbitMQ](eventmesh-connector-rabbitmq) | ✅ | ✅ | | ||
| | [Redis](eventmesh-connector-redis) | ✅ | ✅ | | ||
| | [S3 File](eventmesh-connector-s3) | ⬜ | ✅ | | ||
| | [Slack](eventmesh-connector-slack) | ⬜ | ✅ | | ||
| | [Spring](eventmesh-connector-spring) | ✅ | ✅ | | ||
| | [WeCom](eventmesh-connector-wecom) | ⬜ | ✅ | | ||
| | [WeChat](eventmesh-connector-wechat) | ⬜ | ✅ | | ||
| | More connectors will be added... | N/A | N/A | |
2 changes: 1 addition & 1 deletion
2
...cument/03-connect/07-knative-connector.md → ...cument/03-connect/07-connector-knative.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| # Knative Connector 插件 | ||
| # Knative连接器 | ||
|
|
||
| ## 准备 | ||
|
|
||
|
|
||
35 changes: 35 additions & 0 deletions
35
...rus-plugin-content-docs/current/design-document/03-connect/08-connector-lark.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # Lark连接器 | ||
|
|
||
| ## Lark Sink Server 的配置与启动 | ||
|
|
||
| 使用 eventmesh-connector-lark 下沉事件之前，需要进行 server 的配置。 | ||
| - 请在`/resource/server-config.yml`中自定义`sinkEnable``=`true`/`false`以开启/关闭 sink 功能。 | ||
| - 关于`/resource/sink-config.yml`，在此仅说明`sinkConnectorConfig`下的配置： | ||
| - `connectorName`, 指定 connector 名称 | ||
| - (必需)`appId`, lark 中获取的 appId | ||
| - (必需)`appSecret`, lark 中获取的 appSecret | ||
| - `receiveIdType`，接收 Id 的类型，默认且推荐使用`open_id`。可选 open_id/user_id/union_id/email/chat_id。 | ||
| - (必需)`receiveId`, 接收 Id，需要和`receiveIdType`对应。 | ||
| - `sinkAsync`, 是否异步下沉事件 | ||
| - `maxRetryTimes`, sink 事件失败时，最大重传的次数。默认 3 次。 | ||
| - `retryDelayInMills`, sink 事件失败时，重传事件的时间间隔。默认 1s，单位为毫秒。 | ||
|
|
||
|
|
||
| ## 可下沉飞书的 CLoudEvent | ||
|
|
||
| 使用 eventmesh-connector-lark 下沉事件时，需要在 CloudEvent 中添加对应的 extension filed： | ||
| - 当 key=`templatetype4lark`时，value=`text`/`markdown`，表明该事件的文本类型 | ||
| - 当文本类型为 markdown 时，可以添加 extension：key=`markdownmessagetitle4lark`,value 表明该事件的标题。 | ||
| - 当 key=`atusers4lark`时，value=`id-0,name-0;id-1,name-1`，表明该事件需要`@`某些用户 | ||
| - id 推荐使用**open_id**。 | ||
| - 当文本属于 text 类型时，id 可以是**open_id/union_id/user_id**;当文本属于 markdown 类型时，id 可以是**open_id/user_id**。特别地，当应用类型为[自定义机器人](https://open.feishu.cn/document/ukTMukTMukTM/ucTM5YjL3ETO24yNxkjN)且文本属于 markdown 类型，则仅支持使用**open_id**来`@`用户。 | ||
| - 当文本属于 text 类型且 id 无效时，将利用 name 代替展示；当文本属于 markdown 类型时且 id 无效时，直接抛出异常 (您应该尽量保证 id 的正确性，而 name 则可以考虑省略)。 | ||
| - 当 key=`atall4lark`时，value=`true`/`false`，表明该事件需要`@`所有人。 | ||
|
|
||
|
|
||
| ## 飞书开放平台 API | ||
|
|
||
| 有关该模块涉及到的飞书开放平台 API，请点击以下链接： | ||
| - **发送消息**，请[查看这里](https://open.feishu.cn/document/server-docs/im-v1/message/create?appId=cli_a5e1bc31507ed00c) | ||
| - **text**，请[查看这里](https://open.feishu.cn/document/server-docs/im-v1/message-content-description/create_json#c9e08671) | ||
| - **markdown**，请[查看这里](https://open.feishu.cn/document/common-capabilities/message-card/message-cards-content/using-markdown-tags) |
40 changes: 40 additions & 0 deletions
40
...plugin-content-docs/current/design-document/03-connect/09-connector-rabbitmq.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| # RabbitMQ连接器 | ||
|
|
||
| ## RabbitMQSinkConnector：从 eventmesh 到 rabbitmq。 | ||
|
|
||
| 1. 启动你的 rabbitmq 服务和 eventmesh-runtime。 | ||
| 2. 启用 sinkConnector 并检查 `sink-config.yml`。 | ||
| 3. 向 eventmesh 发送带有在 `pubSubConfig.subject` 中定义的主题消息。 | ||
| ```yaml | ||
| pubSubConfig: | ||
| # 默认端口 10000 | ||
| meshAddress: your.eventmesh.server:10000 | ||
| subject: TopicTest | ||
| idc: FT | ||
| env: PRD | ||
| group: rabbitmqSink | ||
| appId: 5031 | ||
| userName: rabbitmqSinkUser | ||
| passWord: rabbitmqPassWord | ||
| connectorConfig: | ||
| connectorName: rabbitmqSink | ||
| host: your.rabbitmq.server | ||
| port: 5672 | ||
| username: coyrqpyz | ||
| passwd: passwd | ||
| virtualHost: coyrqpyz | ||
| exchangeType: TOPIC | ||
| # 使用内置的 exchangeName 或在连接到 rabbitmq 服务后创建新的 exchangeName。 | ||
| exchangeName: amq.topic | ||
| # 如果在连接之前不存在，rabbitmq 服务将自动创建 routingKey 和 queueName。 | ||
| routingKey: eventmesh | ||
| queueName: eventmesh | ||
| autoAck: true | ||
| ``` | ||
| ## RabbitMQSourceConnector：从 rabbitmq 到 eventmesh。 | ||
| 1. 启动你的 rabbitmq 服务器和 eventmesh-runtime。 | ||
| 2. 启用 sourceConnector 并检查 `source-config.yml`（与 sink-config.yml 基本相同）。 | ||
| 3. 启动你的 RabbitMQConnectorServer，你会在 rabbitmq 服务中找到该channel。 | ||
| 4. 向队列发送一个 cloudevent 消息，然后你将在 eventmesh 中接收到该消息。 |