TDL-23186 Schema automation (#963)

* generated shemas

* update for discovery mode files

* Update format_json.py

* Update format_json.py

* Update get-json-files.py

* generated missing schemas

* Update snapshot-attributes.yml

* remove shared json files

* re-generated after updating from master

* fix ref loop and format issue

* fix format loop

* Update get-json-files.py

* fix for amazon dsp schema

* Update format_json.py

* updated schemas

* handle arrays and anyOf in HTML template

* generated schemas

* Update wootric-latest.md

* remove old files and rename new ones

* sort json

* generated ordered json files

* add list of tables

* convet markdown schemas to json

* Revert "convet markdown schemas to json"

This reverts commit 69e43c8.

* Create convert_md_schemas.py

* Update convert_md_schemas.py

* converted md schemas to json

* Update amplitude-v1-tables.yml

* add option to ignore json schemas

* Update get-table-data-from-md.py

* fixed schema issues

* converted md schemas

* Update convert_md_schemas.py

* updates for parent/child tables

* remove old template

* remove empty foreign keys

* fixed foreign key issues

* added note about nested data structures

* added primary key and replication key icons

* renamed files

* Update metadata_accounts.json

* Revert "Update metadata_accounts.json"

This reverts commit 57fa2f2.

* format json

* fix json

* Revert "fix json"

This reverts commit 8b99a39.

* Revert "format json"

This reverts commit b168d62.

* Revert "Revert "format json""

This reverts commit d6ef036.

* Revert "Revert "fix json""

This reverts commit a153446.

* Update allocations.json

* fix json format issues

* refactor integration data script

* Update check_json_issues.py

* move scripts

* updated paths

* Update get_table_data_from_md.py

* refactored table data script

* Create manual_json_import.py

* cleaned up scripts

* delete file after conversion

* added github actions

* Update import_json_schemas.yml

* fixed workflows

* fixed module name

* Create catalog.json

* Update get_integration_data.py

* fixed workflow

* Automated commit: Generate JSON schemas

* Revert "Automated commit: Generate JSON schemas"

This reverts commit 933d3b7.

* Update get_json_schema_from_catalog.yml

* Automated commit: Generate JSON schemas

* Revert "Automated commit: Generate JSON schemas"

This reverts commit 2616ed7.

* Delete catalog.json

* renamed file

* add foreign keys workflow

* testing

* Revert "testing"

This reverts commit 1f1d937.

* Update generate_foreign_keys.yml

* testing

* Update generate_foreign_keys.yml

* Revert "testing"

This reverts commit 835b5e6.

* replace index with column name

* tsting

* Automated commit: Generate JSON schemas

* Revert "Automated commit: Generate JSON schemas"

This reverts commit 4a1decb.

* Revert "tsting"

This reverts commit 47f8b68.

* Update generate_foreign_keys.yml

* Update import_json_schemas.yml

* fixed files

* fix files

* Create sample-report.json

* TDL-24280 started documenting scripts and workflows

* check in anyof elements

* changed display for arrays

* remove issues file if empty

* Update schemas-table.html

* fixed 3plcentral issues

* Update README.md

* Update README.md

* Update check_json_issues.py

* fix for empty properties

* Update check_table_data.py

* Update generate_foreign_keys.py

* Update get_integration_data.py

* Update get_table_data.py

* remove ignored files from list of issues

* fixed activecampaign issues

* Update get_table_data.py

* removed amazon dsp

* fixed issues

* fixed asana issues

* fixed autopilot issues

* fixed bigcommerce issues

* fixed asana v2

* fixed bing ads v2

* Delete bing-ads-v1-issues.txt

* fixed braintree

* fixed bronto

* fixed chargebee

* fixed chargify

* fixed closeio

* revert to older version for demo

* ignore leads stream

* fixed facebook issues

* fix for schemas without doc links

* fixed clubspeed

* fixed codat

* fixed covid-19 issues

* fixed crossbeam

* Automated commit: Generate foreign keys

* fixed schemas

* Update convert_md_schemas.py

* fixed schemas

* fix deputy issues

* fixed ebay

* fixed eloqua

* fixed freshdesk

* update branch for demo

* fixed pendo issues

* add catalog file for demo

* remove element for demo

* Delete catalog.json

* Update get_json_schema_from_catalog.yml

* remove netsuite schema links

* fixed frontapp issues

* fix for system columns

* Delete fullstory-v1-issues.txt

* fixed github issues

* fixed gitlab issues

* ignore issues

* Update check_table_data.py

* ignore reports for table check

* fixed adwords issues

* replace system column names

* add table templates

* fixed ga360 issues

* fixed google sheets

* Update get_table_data.py

* fixed harvest

* fixed harvest issues

* fixed harvest forecast

* remove not found status for taps with no schemas

* fixed heap issues

* add default value

* fixed helpscout

* fixed ilevel issues

* fixed impact issues

* removed insided

* fixed intercom issues

* fixed invoiced issues

* fixed iterable

* fixed iterable

* fixed jira issues

* fixed klaviyo issues

* Update get_json_files.py

* fixed kustomer issues

* fixed liveperson issues

* fixed looker issues

* fixed lookml

* fixed mailshake

* fixed mambu issues

* fixed marketo

* fixed mixpanel

* fixed teams issues

* fixed onfleet issues

* fixed outbrain

* fixed outreach issues

* fixed pepperjam

* fixed pipedrive

* remove quick-base

* fixed quickbooks

* fixed recharge issue

* fixed recurly issues

* fixed referral-saasquatch

* fixed revinate

* fixed sailthru

* fixed salesforce

* fixed salesforce marketing cloud issues

* fixed selligent

* fixed shiphero

* fixed shippo

* fixed shopify

* fixed slack

* disable table data check for report tables

* fixed snapchat issues

* fixed snapchat issues

* Update check_json_issues.py

* fixed square issues

* fixed stripe issues

* fixed taboola issues

* fixed toggl issues

* fixed trello issues

* fixed twilio issues

* fixed twitter issues

* fixed typeform issues

* fixed ujet issues

* fixed urban-airship issues

* fixed urban-airship issues

* fixed uservoice issues

* fixed wootric issues

* removed workday

* fixed xero issues

* Update get_json_files.py

* fixed yotpo issues

* fixed zendesk issues

* fixed zoom

* fixed zendesk-chat

* renamed files

* added renamed files

* fixed some zuora issues

* fixed google integrations

* changes system for files to ignore

* moved old integration schema files

* fixed hubspot issues

* replaced v0 with v1

* deal with tables not found

* improve script

* fixed issues

* fixed duplicate and broken links

* fixed issues

* updated for frontapp v2

* fixed facebook issues

* Update schemas.html

* fixed hubspot issue

---------

Co-authored-by: github-actions <[email protected]>

.github/workflows/generate_foreign_keys.yml

@@ -3,8 +3,8 @@ on:
   push:
     paths:
       - '**-foreign-keys.yml'
-    branches:
-      - '!master'
+    branches-ignore:
+      - 'master'
 
 jobs:
   generate-foreign-keys:
@@ -23,6 +23,7 @@ jobs:
         with:
           packages: |
             pyyaml
+            pandas
 
       - name: Generate foreign keys
         run: python generate_foreign_keys.py
@@ -38,5 +39,5 @@ jobs:
           git add -A
           git config --global user.email "[email protected]"
           git config --global user.name 'github-actions'
-          git commit -am "Automated commit: Generate JSON schemas"
+          git commit -am "Automated commit: Generate foreign keys"
           git push

.github/workflows/get_json_schema_from_catalog.yml

@@ -24,7 +24,7 @@ on:
         required: true
 
 jobs:
-  generate-changelog:
+  get-json-schemas:
     runs-on: ubuntu-latest
     steps:      
       - name: Get current date
@@ -51,7 +51,7 @@ jobs:
             pyyaml
 
       - name: Get JSON files
-        run: python manual_json_import.py ${{ secrets.GITHUB_TOKEN }} ${{ github.event.inputs.tap }} ${{ github.event.inputs.version }} ${{ github.event.inputs.file }}
+        run: python manual_json_import.py ${{ github.event.inputs.tap }} ${{ github.event.inputs.version }} ${{ github.event.inputs.file }}
         working-directory: ./scripts/json
 
       - name: Check for changes
@@ -74,11 +74,11 @@ jobs:
           branch: ${{ github.event.inputs.branch }}
 
       - name: Create pull request
-        if: steps.get_changes.outputs.changed != 0 && ${{ github.event.inputs.pr }} == true
+        if: steps.get_changes.outputs.changed != 0 && github.event.inputs.pr == 'true'
         uses: repo-sync/pull-request@v2
         with:
           source_branch: ${{ github.event.inputs.branch }}
-          destination_branch: "master"
+          destination_branch: "TDL-23186-json-schema"
           github_token: ${{ secrets.GITHUB_TOKEN }}
           pr_label: "json-schemas-update"
           pr_title: ${{ github.event.inputs.tap }} JSON schema updates ${{ env.date }}

.github/workflows/import_json_schemas.yml

@@ -12,7 +12,7 @@ on:
         required: false
 
 jobs:
-  generate-changelog:
+  import-schemas:
     runs-on: ubuntu-latest
     steps:      
       - name: Get current date
@@ -47,6 +47,7 @@ jobs:
         with:
           packages: |
             requests
+            pyyaml
 
       - name: Get JSON files
         run: python get_json_files.py ${{ secrets.GITHUB_TOKEN }} ${{ github.event.inputs.repo }} ${{ github.event.inputs.branch }}
@@ -62,7 +63,7 @@ jobs:
           git add -A
           git config --global user.email "[email protected]"
           git config --global user.name 'github-actions'
-          git commit -am "Automated commit: Add JSON schemas"
+          git commit -am "Automated commit: Import JSON schemas"
       
       - name: Push changes
         if: steps.get_changes.outputs.changed != 0
@@ -76,7 +77,7 @@ jobs:
         uses: repo-sync/pull-request@v2
         with:
           source_branch: ${{ env.branch }}
-          destination_branch: "master"
+          destination_branch: "TDL-23186-json-schema"
           github_token: ${{ secrets.GITHUB_TOKEN }}
           pr_label: "json-schemas-update"
           pr_title: ${{ github.event.inputs.repo }} JSON schema updates ${{ env.date }}

.gitignore

@@ -13,3 +13,5 @@ Thumbs.db
 *~
 *.scssc
 .cache_ggshield
+__pycache__
+*.pyc