Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE #5108] Abstracting and transforming EventMeshFunction, and implementing FunctionRuntime. #5109

Merged
merged 5 commits into from
Oct 28, 2024

Conversation

cnzakii
Copy link
Contributor

@cnzakii cnzakii commented Sep 24, 2024

Fixes #5108

Motivation

To adapt to the V2 architecture, the EventMeshFunction section needs to be upgraded and transformed. This includes redefining EventMeshFunction, making adaptive modifications to the existing filter and transformer functionalities, and reserving a unified interface for potential new Function features in the future. Based on this, the code for FunctionRuntime will be completed.

Modifications

For the Function module:

  1. A new eventmesh-function module has been added, consolidating the api module and the original filter and transformer modules for easier management.
  2. The EventMeshFunction interface has been defined with reference to java.util.function.Function. The data in Eventmesh is processed through the apply method of the function.
  3. The AbstractEventMeshFunctionChain subclass has been abstracted from EventMeshFunction, defining additional methods for chain processing, such as addFirst, addLast, etc.
  4. The original Transformer interface now inherits from and implements the EventMeshFunction interface, including the apply method for compatibility.

For the FunctionRuntime module:

  1. Referring to ConnectorRuntime, it implements connecting to adminServer, reporting heartbeat and current status to the adminServer, obtaining configuration information from the adminServer, and initializing the connectorService.
  2. A function processing chain is constructed based on the FunctionConfigs in the configuration and is used to process data in ConnectRecord.

Supplement:

Regarding FunctionConfigs in FunctionRuntime, the related configurations and functionalities are as follows:

  1. Users can configure any number of EventMeshFunction in FunctionConfigs according to their needs:

    functionConfigs:
      # filter
      - functionType: filter
        condition:
          source:
            - prefix: "eventmesh."
      # transformer
      - functionType: transformer
        transformerType: template
        valueMap:
          data-name: "$.data.name"
          data-num: "$.data.num"
          event-type: "$.type"
        template: "{\"type\":\"${event-type}\",\"name\":\"${data-name}\",\"no\":${data-num}}"
      # filter
      - functionType: filter
        condition:
          type:
            - suffix: "put"
      # transformer
      - functionType: transformer
        transformerType: constant
        content: "{\"name\":\"jack\",\"no\":123}"

    In the above EventMeshFunction, besides specifying the functionType, other fields are the same as those required by the original Function. For example, the filter requires condition information, which only needs to be added in the YAML file format.

  2. The EventMeshFunction in FunctionConfigs will be built into a function processing chain in sequence:

    input -> filter -> transformer -> filter -> transformer -> output
    

    The data entering the FunctionRuntime will flow along the route above. If the data is filtered, the processing will exit directly.

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Copy link
Contributor

@xwm1992 xwm1992 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Pil0tXia Pil0tXia merged commit 98fbf62 into apache:master Oct 28, 2024
10 checks passed
@cnzakii cnzakii deleted the ospp branch October 28, 2024 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Abstracting and transforming EventMeshFunction, and implementing FunctionRuntime.
3 participants