Skip to content

Commit

Permalink
"Created by snbot"
Browse files Browse the repository at this point in the history
  • Loading branch information
streamnativebot committed Nov 8, 2022
1 parent 7f8eb0b commit 8377f57
Show file tree
Hide file tree
Showing 5 changed files with 108 additions and 133 deletions.
32 changes: 19 additions & 13 deletions .github/workflows/documentbot.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
Expand All @@ -21,13 +20,13 @@
name: Auto Labeling

on:
pull_request_target :
pull_request_target:
types:
- opened
- edited
- labeled
- opened
- edited
- labeled




# A GitHub token created for a PR coming from a fork doesn't have
# 'admin' or 'write' permission (which is required to add labels)
Expand All @@ -39,13 +38,20 @@ jobs:
labeling:
if: ${{ github.repository == 'streamnative/pulsar-io-template' }}
permissions:
pull-requests: write
pull-requests: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- uses: streamnative/github-workflow-libraries/doc-label-check@master
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
label-pattern: '- \[(.*?)\] ?`(.+?)`' # matches '- [x] `label`'
- uses: s4u/[email protected]
with:
servers: '[{"id": "ossrh", "username": "${{ secrets.SONATYPE_USERNAME }}",
"password": "${{ secrets.SONATYPE_PASSWORD }}"}]'
- name: Login to cloudsmith
run: docker login -u="${{ secrets.CLOUDSMITH_USERNAME }}" -p="${{ secrets.CLOUDSMITH_API_KEY
}}" docker.cloudsmith.io
- uses: actions/checkout@v2

- uses: streamnative/github-workflow-libraries/doc-label-check@master
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
label-pattern: '- \[(.*?)\] ?`(.+?)`' # matches '- [x] `label`'

97 changes: 52 additions & 45 deletions .github/workflows/pr-unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,57 +3,64 @@ name: lakehouse mvn build check and unit tests
on:
pull_request:
branches:
- master
- branch-*
- master
- branch-*
push:
branches:
- master
- branch-*
- master
- branch-*

jobs:
build:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v1
- name: Set up JDK 1.8
uses: actions/setup-java@v1
with:
java-version: 1.8

- name: License check
run: mvn -ntp -B license:check

- name: Build cloud package with Maven skipTests
run: mvn clean install -ntp -B -DskipTests -P cloud

- name: Build with Maven skipTests
run: mvn clean install -ntp -B -DskipTests

- name: Style check
run: mvn -ntp -B checkstyle:check

- name: Spotbugs check
run: mvn -ntp -B spotbugs:check

- name: unit test after build
env:
CLOUD_BUCKET_NAME: ${{ secrets.CLOUD_BUCKET_NAME }}
AWS_SECRET_KEY: ${{ secrets.AWS_SECRET_KEY }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
run: mvn test -Pcloud

- name: package surefire artifacts
if: failure()
run: |
rm -rf artifacts
mkdir artifacts
find . -type d -name "*surefire*" -exec cp --parents -R {} artifacts/ \;
zip -r artifacts.zip artifacts
- uses: actions/upload-artifact@master
name: upload surefire-artifacts
if: failure()
with:
name: surefire-artifacts
path: artifacts.zip
- uses: s4u/[email protected]
with:
servers: '[{"id": "ossrh", "username": "${{ secrets.SONATYPE_USERNAME }}",
"password": "${{ secrets.SONATYPE_PASSWORD }}"}]'
- name: Login to cloudsmith
run: docker login -u="${{ secrets.CLOUDSMITH_USERNAME }}" -p="${{ secrets.CLOUDSMITH_API_KEY
}}" docker.cloudsmith.io
- uses: actions/checkout@v1
- name: Set up JDK 1.8
uses: actions/setup-java@v1
with:
java-version: 1.8

- name: License check
run: mvn -ntp -B license:check

- name: Build cloud package with Maven skipTests
run: mvn clean install -ntp -B -DskipTests -P cloud

- name: Build with Maven skipTests
run: mvn clean install -ntp -B -DskipTests

- name: Style check
run: mvn -ntp -B checkstyle:check

- name: Spotbugs check
run: mvn -ntp -B spotbugs:check

- name: unit test after build
env:
CLOUD_BUCKET_NAME: ${{ secrets.CLOUD_BUCKET_NAME }}
AWS_SECRET_KEY: ${{ secrets.AWS_SECRET_KEY }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
run: mvn test -Pcloud

- name: package surefire artifacts
if: failure()
run: |
rm -rf artifacts
mkdir artifacts
find . -type d -name "*surefire*" -exec cp --parents -R {} artifacts/ \;
zip -r artifacts.zip artifacts
- uses: actions/upload-artifact@master
name: upload surefire-artifacts
if: failure()
with:
name: surefire-artifacts
path: artifacts.zip
18 changes: 9 additions & 9 deletions docs/lakehouse-sink.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ To build the Lakehouse sink connector from the source code, follow these steps.

```bash
ls target
pulsar-io-lakehouse-{{connector:version}}.nar
pulsar-io-lakehouse-2.10.1.12.nar
```

# How to configure
Expand Down Expand Up @@ -133,7 +133,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-hudi-pulsar"
],
"archive": "connectors/pulsar-io-hudi-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-hudi-2.10.1.12.nar",
"parallelism": 1,
"configs": {
"type": "hudi",
Expand All @@ -156,7 +156,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-hudi-pulsar"
],
"archive": "connectors/pulsar-io-hudi-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-hudi-2.10.1.12.nar",
"parallelism": 1,
"configs": {
"type": "hudi",
Expand Down Expand Up @@ -185,7 +185,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-iceberg-pulsar"
],
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-2.10.1.12.nar",
"processingGuarantees":"EFFECTIVELY_ONCE",
"configs":{
"type":"iceberg",
Expand Down Expand Up @@ -213,7 +213,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-iceberg-pulsar"
],
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-2.10.1.12.nar",
"processingGuarantees":"EFFECTIVELY_ONCE",
"configs":{
"type":"iceberg",
Expand Down Expand Up @@ -246,7 +246,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-delta-pulsar"
],
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-2.10.1.12.nar",
"processingGuarantees":"EFFECTIVELY_ONCE",
"configs":{
"type":"delta",
Expand All @@ -268,7 +268,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-delta-pulsar"
],
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-2.10.1.12.nar",
"processingGuarantees":"EFFECTIVELY_ONCE",
"configs":{
"type":"delta",
Expand Down Expand Up @@ -339,7 +339,7 @@ This example describes how to use the Lakehouse sink connector to fetch data fro
1. Copy the NAR package to the Pulsar connectors directory.
```
cp pulsar-io-lakehouse-{{connector:version}}.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-{{connector:version}}.nar
cp pulsar-io-lakehouse-2.10.1.12.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-2.10.1.12.nar
```
2. Start Pulsar in standalone mode.
Expand Down Expand Up @@ -372,7 +372,7 @@ This example explains how to create a Lakehouse sink connector in an on-premises
1. Copy the NAR package of the Lakehouse sink connector to the Pulsar connectors directory.
```bash
cp pulsar-io-lakehouse-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-{{connector:version}}.nar
cp pulsar-io-lakehouse-2.10.1.12.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-2.10.1.12.nar
```
2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
Expand Down
10 changes: 5 additions & 5 deletions docs/lakehouse-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ To build the Lakehouse source connector from the source code, follow these steps

```bash
ls target
pulsar-io-lakehouse-{{connector:version}}.nar
pulsar-io-lakehouse-2.10.1.12.nar
```

# How to configure
Expand Down Expand Up @@ -92,7 +92,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"parallelism":1,
"topicName": "delta_source",
"processingGuarantees":"ATLEAST_ONCE",
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-2.10.1.12.nar",
"configs":{
"type":"delta",
"checkpointInterval": 180,
Expand All @@ -117,7 +117,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"parallelism":1,
"topicName": "delta_source",
"processingGuarantees":"ATLEAST_ONCE",
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-2.10.1.12.nar",
"configs":{
"type":"delta",
"checkpointInterval": 180,
Expand Down Expand Up @@ -178,7 +178,7 @@ This example describes how to use the Lakehouse source connector to fetch data f
1. Copy the NAR package to the Pulsar connectors directory.
```
cp pulsar-io-lakehouse-{{connector:version}}.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-{{connector:version}}.nar
cp pulsar-io-lakehouse-2.10.1.12.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-2.10.1.12.nar
```
2. Start Pulsar in standalone mode.
Expand Down Expand Up @@ -209,7 +209,7 @@ This example explains how to create a Lakehouse source connector in an on-premis
1. Copy the NAR package of the Lakehouse source connector to the Pulsar connectors directory.
```
cp pulsar-io-lakehouse-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-{{connector:version}}.nar
cp pulsar-io-lakehouse-2.10.1.12.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-2.10.1.12.nar
```
2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
Expand Down
Loading

0 comments on commit 8377f57

Please sign in to comment.