From 17fa63fd9cf5d52b6c1b6882fdbbfb1f6cb32831 Mon Sep 17 00:00:00 2001 From: "github-classroom[bot]" <66690702+github-classroom[bot]@users.noreply.github.com> Date: Sat, 14 Sep 2024 14:17:42 +0000 Subject: [PATCH 01/23] Setting up GitHub Classroom Feedback From c2273fc26ec679b67706a1ab64c87ae56d6cfac7 Mon Sep 17 00:00:00 2001 From: "github-classroom[bot]" <66690702+github-classroom[bot]@users.noreply.github.com> Date: Sat, 14 Sep 2024 14:17:45 +0000 Subject: [PATCH 02/23] add deadline --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index f0aad2ebc..615a8542f 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,4 @@ +[![Review Assignment Due Date](https://classroom.github.com/assets/deadline-readme-button-22041afd0340ce965d47ae6ef1cefeee28c7c493a6346c4f15d667ab976d596c.svg)](https://classroom.github.com/a/AHFn7Vbn) # Superjoin Hiring Assignment ### Welcome to Superjoin's hiring assignment! 🚀 From 43717ec1fec143e4befa074e7e6e278990bbaf55 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 01:57:59 +0530 Subject: [PATCH 03/23] added video --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 615a8542f..62fb6f1c9 100644 --- a/README.md +++ b/README.md @@ -60,3 +60,5 @@ All the best ✨. ## Developer's Section *Add your video here, and your approach to the problem (optional). Leave some comments for us here if you want, we will be reading this :)* +[Watch the video](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) + From a4eda31f85d23f1545f13577628759cec7a808c8 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 01:59:15 +0530 Subject: [PATCH 04/23] tick marks --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 62fb6f1c9..164d394cd 100644 --- a/README.md +++ b/README.md @@ -45,11 +45,11 @@ Once you're done, make sure you **record a video** showing your project working. We have a checklist at the bottom of this README file, which you should update as your progress with your assignment. It will help us evaluate your project. -- [ ] My code's working just fine! 🥳 -- [ ] I have recorded a video showing it working and embedded it in the README ▶️ -- [ ] I have tested all the normal working cases 😎 +- [x] My code's working just fine! 🥳 +- [x] I have recorded a video showing it working and embedded it in the README ▶️ +- [x] I have tested all the normal working cases 😎 - [ ] I have even solved some edge cases (brownie points) 💪 -- [ ] I added my very planned-out approach to the problem at the end of this README 📜 +- [x] I added my very planned-out approach to the problem at the end of this README 📜 ## Got Questions❓ Feel free to check the discussions tab, you might get some help there. Check out that tab before reaching out to us. Also, did you know, the internet is a great place to explore? 😛 @@ -60,5 +60,7 @@ All the best ✨. ## Developer's Section *Add your video here, and your approach to the problem (optional). Leave some comments for us here if you want, we will be reading this :)* + + [Watch the video](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From 170076aaff15619910835de8b29b575fa2550771 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:09:52 +0530 Subject: [PATCH 05/23] architetcure --- README.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 164d394cd..9e1747bc9 100644 --- a/README.md +++ b/README.md @@ -59,8 +59,24 @@ We're available at techhiring@superjoin.ai for all queries. All the best ✨. ## Developer's Section -*Add your video here, and your approach to the problem (optional). Leave some comments for us here if you want, we will be reading this :)* +# Architectue of the application: +My application is built using Spring Boot and PostgreSQL, providing robust real-time synchronization between multiple pairs of database tables and Google Sheets. The system architecture includes the following key components: +*Real-Time Synchronization:* + +Google Sheets Integration: I use Google Apps Script to send notifications to my backend whenever changes are made in the Google Sheets. This ensures immediate updates and synchronization with the database. +Database Integration: PostgreSQL functions and triggers are employed to notify the backend of any changes detected in the local database. This setup guarantees that updates in the database are promptly reflected in Google Sheets. + +*Dynamic Configuration:* + +Schema Management: The application dynamically constructs schemas for both the database tables and corresponding Google Sheets. This automatic configuration simplifies the setup process and ensures consistency between data sources. +CRUD Operations: My system supports full Create, Read, Update, and Delete (CRUD) operations for both Google Sheets and the PostgreSQL database. It seamlessly handles modifications and maintains synchronization across platforms. +Trigger and Function Management: The application dynamically creates necessary triggers and functions for each table, facilitating automated responses to data changes. + +*Deployment:* + +Local Database: The PostgreSQL database is currently hosted locally. +Remote Access: The Spring Boot application is exposed to the internet using ngrok, allowing external access and interaction with the application. [Watch the video](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From f03e72d9f6919c9f618827275eac5daeb24e8095 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:12:08 +0530 Subject: [PATCH 06/23] formattnig --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 9e1747bc9..80b1ca315 100644 --- a/README.md +++ b/README.md @@ -62,21 +62,21 @@ All the best ✨. # Architectue of the application: My application is built using Spring Boot and PostgreSQL, providing robust real-time synchronization between multiple pairs of database tables and Google Sheets. The system architecture includes the following key components: -*Real-Time Synchronization:* +**Real-Time Synchronization:** -Google Sheets Integration: I use Google Apps Script to send notifications to my backend whenever changes are made in the Google Sheets. This ensures immediate updates and synchronization with the database. -Database Integration: PostgreSQL functions and triggers are employed to notify the backend of any changes detected in the local database. This setup guarantees that updates in the database are promptly reflected in Google Sheets. +*Google Sheets Integration*: I use Google Apps Script to send notifications to my backend whenever changes are made in the Google Sheets. This ensures immediate updates and synchronization with the database. +_Database Integration_: PostgreSQL functions and triggers are employed to notify the backend of any changes detected in the local database. This setup guarantees that updates in the database are promptly reflected in Google Sheets. -*Dynamic Configuration:* +**Dynamic Configuration:** -Schema Management: The application dynamically constructs schemas for both the database tables and corresponding Google Sheets. This automatic configuration simplifies the setup process and ensures consistency between data sources. -CRUD Operations: My system supports full Create, Read, Update, and Delete (CRUD) operations for both Google Sheets and the PostgreSQL database. It seamlessly handles modifications and maintains synchronization across platforms. -Trigger and Function Management: The application dynamically creates necessary triggers and functions for each table, facilitating automated responses to data changes. +_Schema Management_: The application dynamically constructs schemas for both the database tables and corresponding Google Sheets. This automatic configuration simplifies the setup process and ensures consistency between data sources. +_CRUD Operations_: My system supports full Create, Read, Update, and Delete (CRUD) operations for both Google Sheets and the PostgreSQL database. It seamlessly handles modifications and maintains synchronization across platforms. +_Trigger and Function Management_: The application dynamically creates necessary triggers and functions for each table, facilitating automated responses to data changes. -*Deployment:* +**Deployment:** -Local Database: The PostgreSQL database is currently hosted locally. -Remote Access: The Spring Boot application is exposed to the internet using ngrok, allowing external access and interaction with the application. +_Local Database:_ The PostgreSQL database is currently hosted locally. +_Remote Access_: The Spring Boot application is exposed to the internet using ngrok, allowing external access and interaction with the application. [Watch the video](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From 6cb014bfe9fa01f6378eeb7b0b762b8f18ffcd38 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:12:40 +0530 Subject: [PATCH 07/23] formatting --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 80b1ca315..37d5c6cf1 100644 --- a/README.md +++ b/README.md @@ -65,17 +65,21 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- **Real-Time Synchronization:** *Google Sheets Integration*: I use Google Apps Script to send notifications to my backend whenever changes are made in the Google Sheets. This ensures immediate updates and synchronization with the database. + _Database Integration_: PostgreSQL functions and triggers are employed to notify the backend of any changes detected in the local database. This setup guarantees that updates in the database are promptly reflected in Google Sheets. **Dynamic Configuration:** _Schema Management_: The application dynamically constructs schemas for both the database tables and corresponding Google Sheets. This automatic configuration simplifies the setup process and ensures consistency between data sources. + _CRUD Operations_: My system supports full Create, Read, Update, and Delete (CRUD) operations for both Google Sheets and the PostgreSQL database. It seamlessly handles modifications and maintains synchronization across platforms. + _Trigger and Function Management_: The application dynamically creates necessary triggers and functions for each table, facilitating automated responses to data changes. **Deployment:** _Local Database:_ The PostgreSQL database is currently hosted locally. + _Remote Access_: The Spring Boot application is exposed to the internet using ngrok, allowing external access and interaction with the application. [Watch the video](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From 8bec12d11bb4a89c8e6be5a8f478b793a5ce0d8d Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:13:57 +0530 Subject: [PATCH 08/23] formatiing --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 37d5c6cf1..98c4f1b49 100644 --- a/README.md +++ b/README.md @@ -62,25 +62,25 @@ All the best ✨. # Architectue of the application: My application is built using Spring Boot and PostgreSQL, providing robust real-time synchronization between multiple pairs of database tables and Google Sheets. The system architecture includes the following key components: -**Real-Time Synchronization:** +* **Real-Time Synchronization:** -*Google Sheets Integration*: I use Google Apps Script to send notifications to my backend whenever changes are made in the Google Sheets. This ensures immediate updates and synchronization with the database. +* *Google Sheets Integration*: I use Google Apps Script to send notifications to my backend whenever changes are made in the Google Sheets. This ensures immediate updates and synchronization with the database. -_Database Integration_: PostgreSQL functions and triggers are employed to notify the backend of any changes detected in the local database. This setup guarantees that updates in the database are promptly reflected in Google Sheets. +* _Database Integration_: PostgreSQL functions and triggers are employed to notify the backend of any changes detected in the local database. This setup guarantees that updates in the database are promptly reflected in Google Sheets. **Dynamic Configuration:** -_Schema Management_: The application dynamically constructs schemas for both the database tables and corresponding Google Sheets. This automatic configuration simplifies the setup process and ensures consistency between data sources. +* _Schema Management_: The application dynamically constructs schemas for both the database tables and corresponding Google Sheets. This automatic configuration simplifies the setup process and ensures consistency between data sources. -_CRUD Operations_: My system supports full Create, Read, Update, and Delete (CRUD) operations for both Google Sheets and the PostgreSQL database. It seamlessly handles modifications and maintains synchronization across platforms. +* _CRUD Operations_: My system supports full Create, Read, Update, and Delete (CRUD) operations for both Google Sheets and the PostgreSQL database. It seamlessly handles modifications and maintains synchronization across platforms. -_Trigger and Function Management_: The application dynamically creates necessary triggers and functions for each table, facilitating automated responses to data changes. +* _Trigger and Function Management_: The application dynamically creates necessary triggers and functions for each table, facilitating automated responses to data changes. **Deployment:** -_Local Database:_ The PostgreSQL database is currently hosted locally. +* _Local Database:_ The PostgreSQL database is currently hosted locally. -_Remote Access_: The Spring Boot application is exposed to the internet using ngrok, allowing external access and interaction with the application. +* _Remote Access_: The Spring Boot application is exposed to the internet using ngrok, allowing external access and interaction with the application. [Watch the video](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From 72bb49e5319387281fac66ee2039685345ba967e Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:27:22 +0530 Subject: [PATCH 09/23] User setup --- README.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 98c4f1b49..561c4f40e 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ We're available at techhiring@superjoin.ai for all queries. All the best ✨. ## Developer's Section -# Architectue of the application: +### Architectue of the application: My application is built using Spring Boot and PostgreSQL, providing robust real-time synchronization between multiple pairs of database tables and Google Sheets. The system architecture includes the following key components: * **Real-Time Synchronization:** @@ -82,5 +82,36 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- * _Remote Access_: The Spring Boot application is exposed to the internet using ngrok, allowing external access and interaction with the application. +### Requirements for the Setup: + +* **Java Development Kit (JDK):** + +- Version 11 or newer, as Spring Boot typically supports recent LTS versions. + +* **PostgreSQL** + +* **ngrok** + - once ngrok has been setup run the following command: + ``` + ngrok http 8080 + ``` + +### User-Setup: + +* The user has to make a Google Sheet and change access from "Restricted" to "Anyone with the link" and set permissions to "Editor" +* When the user runs the application, they will be prompted with a question if they want to create a new table. If they do want to then they must type yes, else the application will still be running just listening to notifiations. +* When they type yes, they will be asked to enter the sheet id. Then they must enter the google sheet's url +* Then they will be prompted to enter the table name, primary key and column names of the table. +* once the table is created on both the google sheets and the database, copy the App Script given below. +* In the google sheets window click on Extentions > App Script. +* Then paste the Appscrpt there. +* then click on the clock like button on the left +* add a new trigger + - set the function to "sendnotification" and "notify me immediately" + - save the trigger +* now the user can use his google sheets and database and synchronisation will start. + + + [Watch the video](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From 8c9e2d9dacb9837c663fb00135331644f8d3b778 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:28:58 +0530 Subject: [PATCH 10/23] app script --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 561c4f40e..f51da5ac2 100644 --- a/README.md +++ b/README.md @@ -102,7 +102,8 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- * When the user runs the application, they will be prompted with a question if they want to create a new table. If they do want to then they must type yes, else the application will still be running just listening to notifiations. * When they type yes, they will be asked to enter the sheet id. Then they must enter the google sheet's url * Then they will be prompted to enter the table name, primary key and column names of the table. -* once the table is created on both the google sheets and the database, copy the App Script given below. +* once the table is created on both the google sheets and the database, copy the App Script given in this repository. + ``` * In the google sheets window click on Extentions > App Script. * Then paste the Appscrpt there. * then click on the clock like button on the left From 54ec6baee350cd28d5956db4b29647c629466b15 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:29:20 +0530 Subject: [PATCH 11/23] appscrpt --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index f51da5ac2..1719adfee 100644 --- a/README.md +++ b/README.md @@ -103,7 +103,6 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- * When they type yes, they will be asked to enter the sheet id. Then they must enter the google sheet's url * Then they will be prompted to enter the table name, primary key and column names of the table. * once the table is created on both the google sheets and the database, copy the App Script given in this repository. - ``` * In the google sheets window click on Extentions > App Script. * Then paste the Appscrpt there. * then click on the clock like button on the left From d92b4bb7518896f20133e4383429a1cade4702bb Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:33:09 +0530 Subject: [PATCH 12/23] aplication setup --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1719adfee..5411e428f 100644 --- a/README.md +++ b/README.md @@ -95,9 +95,9 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- ``` ngrok http 8080 ``` - +### Application-Setup: +* Visit google marketplace > search for sheets > click on the google sheets api and click enable > click on credentails and add a service account > click on the service account and select keys > create a new key, download it and put the JSON file into the directory right outside the src directory in the project. ### User-Setup: - * The user has to make a Google Sheet and change access from "Restricted" to "Anyone with the link" and set permissions to "Editor" * When the user runs the application, they will be prompted with a question if they want to create a new table. If they do want to then they must type yes, else the application will still be running just listening to notifiations. * When they type yes, they will be asked to enter the sheet id. Then they must enter the google sheet's url From a333e43feeaaca093fcf30a7ad0da8507fd1d8ec Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:33:49 +0530 Subject: [PATCH 13/23] formatnig --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5411e428f..c46b6f3a1 100644 --- a/README.md +++ b/README.md @@ -113,5 +113,5 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- -[Watch the video](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) +[**Watch the video**](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From a415dfb5b51fb455536c473ac0411fcdaaa84502 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:34:12 +0530 Subject: [PATCH 14/23] video --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c46b6f3a1..ee03190d5 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,6 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- * now the user can use his google sheets and database and synchronisation will start. - +### Video: [**Watch the video**](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From 36347a39d300c560aa724dc91d6088b2fa723f1c Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:43:09 +0530 Subject: [PATCH 15/23] future scope --- README.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/README.md b/README.md index ee03190d5..8f85cf9ad 100644 --- a/README.md +++ b/README.md @@ -111,6 +111,25 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- - save the trigger * now the user can use his google sheets and database and synchronisation will start. +### Future Scope: + +To enhance the scalability and efficiency of the application, several improvements are planned: + +* **Apache Kafka Integration:** + + - _High-Frequency Requests_: Implementing Kafka will enable the application to handle high-frequency requests by storing all requests in Kafka topics. This approach will help prevent the loss of requests and ensure that no data is missed. + - _Topic Partitioning:_ By partitioning Kafka topics, the application will benefit from faster bulk data management, as partitions allow for parallel processing and better performance. +* **Conflict Management:** + + - _Semaphores vs. Locks_: For managing conflicts when multiple users edit the same table, the application will use semaphores or locks. Given the design of the application, I would prefer semaphores for their simpler implementation. + +* **Global Scaling:** + + - _YugaByte Database:_ To ensure global scalability, the application will leverage YugaByte’s database solution. The paid version of YugaByte supports global data replication, enabling seamless data distribution and high availability across different geographical locations. + + +These enhancements will provide robust handling of high-frequency data requests, efficient conflict management, and scalable data distribution, preparing the application for future growth and expanded use cases. + ### Video: [**Watch the video**](https://drive.google.com/file/d/1ItHfXC3zhYEcZADYyoaXU_pSkeHIPba5/view?usp=sharing) From 057f04acd76737237349806b90b83aaf32516377 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:45:09 +0530 Subject: [PATCH 16/23] app setup --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 8f85cf9ad..874ff9566 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- * **Java Development Kit (JDK):** -- Version 11 or newer, as Spring Boot typically supports recent LTS versions. + - Version 11 or newer, as Spring Boot typically supports recent LTS versions. * **PostgreSQL** @@ -97,6 +97,8 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- ``` ### Application-Setup: * Visit google marketplace > search for sheets > click on the google sheets api and click enable > click on credentails and add a service account > click on the service account and select keys > create a new key, download it and put the JSON file into the directory right outside the src directory in the project. +* in the application.properties file enter your postgres database name in the url and your postgres username and password. + ### User-Setup: * The user has to make a Google Sheet and change access from "Restricted" to "Anyone with the link" and set permissions to "Editor" * When the user runs the application, they will be prompted with a question if they want to create a new table. If they do want to then they must type yes, else the application will still be running just listening to notifiations. From 98fe0275f767bfb470bce9324f5b506396b5a06a Mon Sep 17 00:00:00 2001 From: Joel Renjith Date: Mon, 16 Sep 2024 21:44:15 +0530 Subject: [PATCH 17/23] Initial commit --- .gitignore | 33 +++ .mvn/wrapper/maven-wrapper.properties | 19 ++ credentials.json | 13 + mvnw | 259 ++++++++++++++++++ mvnw.cmd | 149 ++++++++++ pom.xml | 161 +++++++++++ .../example/yuga/start/StartApplication.java | 77 ++++++ .../controller/NotificationController.java | 31 +++ .../start/controller/NotificationPayload.java | 76 +++++ .../yuga/start/gsheet/GoogleSheetsConfig.java | 51 ++++ .../yuga/start/gsheet/SheetsService.java | 242 ++++++++++++++++ .../start/repos/DynamicTableRepository.java | 59 ++++ .../start/repos/SchemaRegistryRepository.java | 129 +++++++++ .../yuga/start/security/SimpleCORSFilter.java | 34 +++ .../start/service/DynamicTableService.java | 214 +++++++++++++++ .../service/PostgresNotificationService.java | 87 ++++++ .../start/service/SchemaRegistryService.java | 46 ++++ .../start/service/SqlNotificationPayload.java | 102 +++++++ src/main/resources/application.properties | 26 ++ .../yuga/start/StartApplicationTests.java | 13 + .../start1/start/StartApplicationTests.java | 13 + 21 files changed, 1834 insertions(+) create mode 100644 .gitignore create mode 100644 .mvn/wrapper/maven-wrapper.properties create mode 100644 credentials.json create mode 100644 mvnw create mode 100644 mvnw.cmd create mode 100644 pom.xml create mode 100644 src/main/java/com/example/yuga/start/StartApplication.java create mode 100644 src/main/java/com/example/yuga/start/controller/NotificationController.java create mode 100644 src/main/java/com/example/yuga/start/controller/NotificationPayload.java create mode 100644 src/main/java/com/example/yuga/start/gsheet/GoogleSheetsConfig.java create mode 100644 src/main/java/com/example/yuga/start/gsheet/SheetsService.java create mode 100644 src/main/java/com/example/yuga/start/repos/DynamicTableRepository.java create mode 100644 src/main/java/com/example/yuga/start/repos/SchemaRegistryRepository.java create mode 100644 src/main/java/com/example/yuga/start/security/SimpleCORSFilter.java create mode 100644 src/main/java/com/example/yuga/start/service/DynamicTableService.java create mode 100644 src/main/java/com/example/yuga/start/service/PostgresNotificationService.java create mode 100644 src/main/java/com/example/yuga/start/service/SchemaRegistryService.java create mode 100644 src/main/java/com/example/yuga/start/service/SqlNotificationPayload.java create mode 100644 src/main/resources/application.properties create mode 100644 src/test/java/com/example/yuga/start/StartApplicationTests.java create mode 100644 src/test/java/com/example/yuga/start1/start/StartApplicationTests.java diff --git a/.gitignore b/.gitignore new file mode 100644 index 000000000..549e00a2a --- /dev/null +++ b/.gitignore @@ -0,0 +1,33 @@ +HELP.md +target/ +!.mvn/wrapper/maven-wrapper.jar +!**/src/main/**/target/ +!**/src/test/**/target/ + +### STS ### +.apt_generated +.classpath +.factorypath +.project +.settings +.springBeans +.sts4-cache + +### IntelliJ IDEA ### +.idea +*.iws +*.iml +*.ipr + +### NetBeans ### +/nbproject/private/ +/nbbuild/ +/dist/ +/nbdist/ +/.nb-gradle/ +build/ +!**/src/main/**/build/ +!**/src/test/**/build/ + +### VS Code ### +.vscode/ diff --git a/.mvn/wrapper/maven-wrapper.properties b/.mvn/wrapper/maven-wrapper.properties new file mode 100644 index 000000000..d58dfb70b --- /dev/null +++ b/.mvn/wrapper/maven-wrapper.properties @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +wrapperVersion=3.3.2 +distributionType=only-script +distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.9/apache-maven-3.9.9-bin.zip diff --git a/credentials.json b/credentials.json new file mode 100644 index 000000000..e140a8473 --- /dev/null +++ b/credentials.json @@ -0,0 +1,13 @@ +{ + "type": "service_account", + "project_id": "gsheetssqlmap", + "private_key_id": "39798df6b3a4ffd23912f9a0e85590a145e39671", + "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCq8mNIvQQE8SM5\nuotOOr7RBjqpctGMlH4DRp5FcqpIz5FNla1AH3s5MIvEEo6L769Exdd+wtAnxdsF\nHaBZxHi58xRsyxkWFNPliQR8B4dqaQM1TxfJ8ocH0IUV/paSNa6uCLnucRMYw/Pu\nZb2kIocdmJSXBvl/BhHm6xNOvai8YOMHcN0fbN5sKCW+Wi3Vyd/kWLeYPn6FHj4T\nXuca6tBlbSALPDoVbcMwpdSfMZvjQ0jvxcZHkvOsfEKCeHxP46ylfkddmfaXywTT\nR0eEi64asFcexGGENE8RxqIVa68T3pWfPvxzWAr16KaVKzCutKIzBOFh636aB1/s\nep9AkSzpAgMBAAECggEAEwyq//FxQ9g3ydB2AuAUwMACFD0raRAomTaav0gzfeTk\njynjajJ/5av5XT6IpEMvy7kExjdRISeBHEk0g+BbKDSUdXmt+FoWkVzzExu3LUZZ\nwUxorRxRzA8Cb6Sk1JL3HeVyy8WNwFwgwBOWHhbraeLFhy1zquIrJ0mRp5wGtEQk\ng9JCY2RkTMH0Ro040UEgdo2oFo+61BMLSR9nnJYdP/0GZ6ooN+OJVEx+MhjDS37f\niJq7iN2MVd0I113SbRQGysmVz781yCoN7TcfBSO4qPX1Z1mMdm3wyv3K9e3afCgC\nS8aR8ZyG6YA1vW8GlLSVcA+pxrZ+GBRNR3hBaBCvUQKBgQDcpIv6s3MQZxbaCBoN\n93L4mmREQ3+/WyK+AqHcfvzrccw5OUkJDRfA3PN+7y39pYSoXGkJdu16HoFBQM8f\n6CgDEGt3iIDQ4VcNupWHBFvu8C8x/Pfk3N7LWqwewC0fzvGZhJbIu45shKdMyeqD\nYPzq8SShG0rfdxBfQpPKWGkhuQKBgQDGVyi5H5s5RhrTIvAz5wRkoSRQwro4TXTm\nV1sUpLqB3XFzpA7zAPl9rDwc1h+EB5cBzBmizCpXqBdUaCYh4e1CRwbdKJunolRK\njKLvANDKHAcRODeXNjll8I/ekKdFlQOu/5ZZBGlAbY/GRUyw5+8fIzxWgQ+ZVkuq\nhfq6loW8sQKBgQDCI+Kq/7Sx7RNVWyWZvm4iQq8QxZelBwqlx+kl+3KNtD6u5zHV\nTfjlkUSLHucHtNjpzw347vT7q9WtVdopx/tU3uDAd5JouK4BfXhOKc6+igXs5Ro1\n/QXoRmsGT7AkudEpdFm8XBNwdrRCQ7QeT3ubOf0x6LLTt47nUsx5kZtHuQKBgA5W\nBHKFD1IveAbk0pndy9p+L0LLIGyrZJS1oba3RWfXmriyFgaIfoAP7/RhobuIVOjj\nWMMEoNbfJniKYp82VlDhE2Tu53lRaWhODATceTrylawv9Qyv9awhnDSJHh4QbEi8\n3qAminkGskqxfZ6X1RfaEyVphLgfna6AqqPL93SxAoGAcatEfglIlVeHYkl4aUV4\nBZvRb35VJc3HTsXfeN0Bnk2Kue46cF3IEgKlXHz/cxhStFPiGRj2fz7LsdZzWPbZ\nmSjSXP/mPycMcYR55SMHhWI95BXDxaC6Eg8N9qP+9TbPm2c8si2KRYa2kbCYfETX\ntq4kjFWG/Z27jKWWi3+uttg=\n-----END PRIVATE KEY-----\n", + "client_email": "gmapsheet@gsheetssqlmap.iam.gserviceaccount.com", + "client_id": "108264969693417699931", + "auth_uri": "https://accounts.google.com/o/oauth2/auth", + "token_uri": "https://oauth2.googleapis.com/token", + "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", + "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gmapsheet%40gsheetssqlmap.iam.gserviceaccount.com", + "universe_domain": "googleapis.com" +} diff --git a/mvnw b/mvnw new file mode 100644 index 000000000..19529ddf8 --- /dev/null +++ b/mvnw @@ -0,0 +1,259 @@ +#!/bin/sh +# ---------------------------------------------------------------------------- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# ---------------------------------------------------------------------------- + +# ---------------------------------------------------------------------------- +# Apache Maven Wrapper startup batch script, version 3.3.2 +# +# Optional ENV vars +# ----------------- +# JAVA_HOME - location of a JDK home dir, required when download maven via java source +# MVNW_REPOURL - repo url base for downloading maven distribution +# MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven +# MVNW_VERBOSE - true: enable verbose log; debug: trace the mvnw script; others: silence the output +# ---------------------------------------------------------------------------- + +set -euf +[ "${MVNW_VERBOSE-}" != debug ] || set -x + +# OS specific support. +native_path() { printf %s\\n "$1"; } +case "$(uname)" in +CYGWIN* | MINGW*) + [ -z "${JAVA_HOME-}" ] || JAVA_HOME="$(cygpath --unix "$JAVA_HOME")" + native_path() { cygpath --path --windows "$1"; } + ;; +esac + +# set JAVACMD and JAVACCMD +set_java_home() { + # For Cygwin and MinGW, ensure paths are in Unix format before anything is touched + if [ -n "${JAVA_HOME-}" ]; then + if [ -x "$JAVA_HOME/jre/sh/java" ]; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + JAVACCMD="$JAVA_HOME/jre/sh/javac" + else + JAVACMD="$JAVA_HOME/bin/java" + JAVACCMD="$JAVA_HOME/bin/javac" + + if [ ! -x "$JAVACMD" ] || [ ! -x "$JAVACCMD" ]; then + echo "The JAVA_HOME environment variable is not defined correctly, so mvnw cannot run." >&2 + echo "JAVA_HOME is set to \"$JAVA_HOME\", but \"\$JAVA_HOME/bin/java\" or \"\$JAVA_HOME/bin/javac\" does not exist." >&2 + return 1 + fi + fi + else + JAVACMD="$( + 'set' +e + 'unset' -f command 2>/dev/null + 'command' -v java + )" || : + JAVACCMD="$( + 'set' +e + 'unset' -f command 2>/dev/null + 'command' -v javac + )" || : + + if [ ! -x "${JAVACMD-}" ] || [ ! -x "${JAVACCMD-}" ]; then + echo "The java/javac command does not exist in PATH nor is JAVA_HOME set, so mvnw cannot run." >&2 + return 1 + fi + fi +} + +# hash string like Java String::hashCode +hash_string() { + str="${1:-}" h=0 + while [ -n "$str" ]; do + char="${str%"${str#?}"}" + h=$(((h * 31 + $(LC_CTYPE=C printf %d "'$char")) % 4294967296)) + str="${str#?}" + done + printf %x\\n $h +} + +verbose() { :; } +[ "${MVNW_VERBOSE-}" != true ] || verbose() { printf %s\\n "${1-}"; } + +die() { + printf %s\\n "$1" >&2 + exit 1 +} + +trim() { + # MWRAPPER-139: + # Trims trailing and leading whitespace, carriage returns, tabs, and linefeeds. + # Needed for removing poorly interpreted newline sequences when running in more + # exotic environments such as mingw bash on Windows. + printf "%s" "${1}" | tr -d '[:space:]' +} + +# parse distributionUrl and optional distributionSha256Sum, requires .mvn/wrapper/maven-wrapper.properties +while IFS="=" read -r key value; do + case "${key-}" in + distributionUrl) distributionUrl=$(trim "${value-}") ;; + distributionSha256Sum) distributionSha256Sum=$(trim "${value-}") ;; + esac +done <"${0%/*}/.mvn/wrapper/maven-wrapper.properties" +[ -n "${distributionUrl-}" ] || die "cannot read distributionUrl property in ${0%/*}/.mvn/wrapper/maven-wrapper.properties" + +case "${distributionUrl##*/}" in +maven-mvnd-*bin.*) + MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/ + case "${PROCESSOR_ARCHITECTURE-}${PROCESSOR_ARCHITEW6432-}:$(uname -a)" in + *AMD64:CYGWIN* | *AMD64:MINGW*) distributionPlatform=windows-amd64 ;; + :Darwin*x86_64) distributionPlatform=darwin-amd64 ;; + :Darwin*arm64) distributionPlatform=darwin-aarch64 ;; + :Linux*x86_64*) distributionPlatform=linux-amd64 ;; + *) + echo "Cannot detect native platform for mvnd on $(uname)-$(uname -m), use pure java version" >&2 + distributionPlatform=linux-amd64 + ;; + esac + distributionUrl="${distributionUrl%-bin.*}-$distributionPlatform.zip" + ;; +maven-mvnd-*) MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/ ;; +*) MVN_CMD="mvn${0##*/mvnw}" _MVNW_REPO_PATTERN=/org/apache/maven/ ;; +esac + +# apply MVNW_REPOURL and calculate MAVEN_HOME +# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-,maven-mvnd--}/ +[ -z "${MVNW_REPOURL-}" ] || distributionUrl="$MVNW_REPOURL$_MVNW_REPO_PATTERN${distributionUrl#*"$_MVNW_REPO_PATTERN"}" +distributionUrlName="${distributionUrl##*/}" +distributionUrlNameMain="${distributionUrlName%.*}" +distributionUrlNameMain="${distributionUrlNameMain%-bin}" +MAVEN_USER_HOME="${MAVEN_USER_HOME:-${HOME}/.m2}" +MAVEN_HOME="${MAVEN_USER_HOME}/wrapper/dists/${distributionUrlNameMain-}/$(hash_string "$distributionUrl")" + +exec_maven() { + unset MVNW_VERBOSE MVNW_USERNAME MVNW_PASSWORD MVNW_REPOURL || : + exec "$MAVEN_HOME/bin/$MVN_CMD" "$@" || die "cannot exec $MAVEN_HOME/bin/$MVN_CMD" +} + +if [ -d "$MAVEN_HOME" ]; then + verbose "found existing MAVEN_HOME at $MAVEN_HOME" + exec_maven "$@" +fi + +case "${distributionUrl-}" in +*?-bin.zip | *?maven-mvnd-?*-?*.zip) ;; +*) die "distributionUrl is not valid, must match *-bin.zip or maven-mvnd-*.zip, but found '${distributionUrl-}'" ;; +esac + +# prepare tmp dir +if TMP_DOWNLOAD_DIR="$(mktemp -d)" && [ -d "$TMP_DOWNLOAD_DIR" ]; then + clean() { rm -rf -- "$TMP_DOWNLOAD_DIR"; } + trap clean HUP INT TERM EXIT +else + die "cannot create temp dir" +fi + +mkdir -p -- "${MAVEN_HOME%/*}" + +# Download and Install Apache Maven +verbose "Couldn't find MAVEN_HOME, downloading and installing it ..." +verbose "Downloading from: $distributionUrl" +verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName" + +# select .zip or .tar.gz +if ! command -v unzip >/dev/null; then + distributionUrl="${distributionUrl%.zip}.tar.gz" + distributionUrlName="${distributionUrl##*/}" +fi + +# verbose opt +__MVNW_QUIET_WGET=--quiet __MVNW_QUIET_CURL=--silent __MVNW_QUIET_UNZIP=-q __MVNW_QUIET_TAR='' +[ "${MVNW_VERBOSE-}" != true ] || __MVNW_QUIET_WGET='' __MVNW_QUIET_CURL='' __MVNW_QUIET_UNZIP='' __MVNW_QUIET_TAR=v + +# normalize http auth +case "${MVNW_PASSWORD:+has-password}" in +'') MVNW_USERNAME='' MVNW_PASSWORD='' ;; +has-password) [ -n "${MVNW_USERNAME-}" ] || MVNW_USERNAME='' MVNW_PASSWORD='' ;; +esac + +if [ -z "${MVNW_USERNAME-}" ] && command -v wget >/dev/null; then + verbose "Found wget ... using wget" + wget ${__MVNW_QUIET_WGET:+"$__MVNW_QUIET_WGET"} "$distributionUrl" -O "$TMP_DOWNLOAD_DIR/$distributionUrlName" || die "wget: Failed to fetch $distributionUrl" +elif [ -z "${MVNW_USERNAME-}" ] && command -v curl >/dev/null; then + verbose "Found curl ... using curl" + curl ${__MVNW_QUIET_CURL:+"$__MVNW_QUIET_CURL"} -f -L -o "$TMP_DOWNLOAD_DIR/$distributionUrlName" "$distributionUrl" || die "curl: Failed to fetch $distributionUrl" +elif set_java_home; then + verbose "Falling back to use Java to download" + javaSource="$TMP_DOWNLOAD_DIR/Downloader.java" + targetZip="$TMP_DOWNLOAD_DIR/$distributionUrlName" + cat >"$javaSource" <<-END + public class Downloader extends java.net.Authenticator + { + protected java.net.PasswordAuthentication getPasswordAuthentication() + { + return new java.net.PasswordAuthentication( System.getenv( "MVNW_USERNAME" ), System.getenv( "MVNW_PASSWORD" ).toCharArray() ); + } + public static void main( String[] args ) throws Exception + { + setDefault( new Downloader() ); + java.nio.file.Files.copy( java.net.URI.create( args[0] ).toURL().openStream(), java.nio.file.Paths.get( args[1] ).toAbsolutePath().normalize() ); + } + } + END + # For Cygwin/MinGW, switch paths to Windows format before running javac and java + verbose " - Compiling Downloader.java ..." + "$(native_path "$JAVACCMD")" "$(native_path "$javaSource")" || die "Failed to compile Downloader.java" + verbose " - Running Downloader.java ..." + "$(native_path "$JAVACMD")" -cp "$(native_path "$TMP_DOWNLOAD_DIR")" Downloader "$distributionUrl" "$(native_path "$targetZip")" +fi + +# If specified, validate the SHA-256 sum of the Maven distribution zip file +if [ -n "${distributionSha256Sum-}" ]; then + distributionSha256Result=false + if [ "$MVN_CMD" = mvnd.sh ]; then + echo "Checksum validation is not supported for maven-mvnd." >&2 + echo "Please disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2 + exit 1 + elif command -v sha256sum >/dev/null; then + if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | sha256sum -c >/dev/null 2>&1; then + distributionSha256Result=true + fi + elif command -v shasum >/dev/null; then + if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | shasum -a 256 -c >/dev/null 2>&1; then + distributionSha256Result=true + fi + else + echo "Checksum validation was requested but neither 'sha256sum' or 'shasum' are available." >&2 + echo "Please install either command, or disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2 + exit 1 + fi + if [ $distributionSha256Result = false ]; then + echo "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised." >&2 + echo "If you updated your Maven version, you need to update the specified distributionSha256Sum property." >&2 + exit 1 + fi +fi + +# unzip and move +if command -v unzip >/dev/null; then + unzip ${__MVNW_QUIET_UNZIP:+"$__MVNW_QUIET_UNZIP"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -d "$TMP_DOWNLOAD_DIR" || die "failed to unzip" +else + tar xzf${__MVNW_QUIET_TAR:+"$__MVNW_QUIET_TAR"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -C "$TMP_DOWNLOAD_DIR" || die "failed to untar" +fi +printf %s\\n "$distributionUrl" >"$TMP_DOWNLOAD_DIR/$distributionUrlNameMain/mvnw.url" +mv -- "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain" "$MAVEN_HOME" || [ -d "$MAVEN_HOME" ] || die "fail to move MAVEN_HOME" + +clean || : +exec_maven "$@" diff --git a/mvnw.cmd b/mvnw.cmd new file mode 100644 index 000000000..249bdf382 --- /dev/null +++ b/mvnw.cmd @@ -0,0 +1,149 @@ +<# : batch portion +@REM ---------------------------------------------------------------------------- +@REM Licensed to the Apache Software Foundation (ASF) under one +@REM or more contributor license agreements. See the NOTICE file +@REM distributed with this work for additional information +@REM regarding copyright ownership. The ASF licenses this file +@REM to you under the Apache License, Version 2.0 (the +@REM "License"); you may not use this file except in compliance +@REM with the License. You may obtain a copy of the License at +@REM +@REM http://www.apache.org/licenses/LICENSE-2.0 +@REM +@REM Unless required by applicable law or agreed to in writing, +@REM software distributed under the License is distributed on an +@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +@REM KIND, either express or implied. See the License for the +@REM specific language governing permissions and limitations +@REM under the License. +@REM ---------------------------------------------------------------------------- + +@REM ---------------------------------------------------------------------------- +@REM Apache Maven Wrapper startup batch script, version 3.3.2 +@REM +@REM Optional ENV vars +@REM MVNW_REPOURL - repo url base for downloading maven distribution +@REM MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven +@REM MVNW_VERBOSE - true: enable verbose log; others: silence the output +@REM ---------------------------------------------------------------------------- + +@IF "%__MVNW_ARG0_NAME__%"=="" (SET __MVNW_ARG0_NAME__=%~nx0) +@SET __MVNW_CMD__= +@SET __MVNW_ERROR__= +@SET __MVNW_PSMODULEP_SAVE=%PSModulePath% +@SET PSModulePath= +@FOR /F "usebackq tokens=1* delims==" %%A IN (`powershell -noprofile "& {$scriptDir='%~dp0'; $script='%__MVNW_ARG0_NAME__%'; icm -ScriptBlock ([Scriptblock]::Create((Get-Content -Raw '%~f0'))) -NoNewScope}"`) DO @( + IF "%%A"=="MVN_CMD" (set __MVNW_CMD__=%%B) ELSE IF "%%B"=="" (echo %%A) ELSE (echo %%A=%%B) +) +@SET PSModulePath=%__MVNW_PSMODULEP_SAVE% +@SET __MVNW_PSMODULEP_SAVE= +@SET __MVNW_ARG0_NAME__= +@SET MVNW_USERNAME= +@SET MVNW_PASSWORD= +@IF NOT "%__MVNW_CMD__%"=="" (%__MVNW_CMD__% %*) +@echo Cannot start maven from wrapper >&2 && exit /b 1 +@GOTO :EOF +: end batch / begin powershell #> + +$ErrorActionPreference = "Stop" +if ($env:MVNW_VERBOSE -eq "true") { + $VerbosePreference = "Continue" +} + +# calculate distributionUrl, requires .mvn/wrapper/maven-wrapper.properties +$distributionUrl = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionUrl +if (!$distributionUrl) { + Write-Error "cannot read distributionUrl property in $scriptDir/.mvn/wrapper/maven-wrapper.properties" +} + +switch -wildcard -casesensitive ( $($distributionUrl -replace '^.*/','') ) { + "maven-mvnd-*" { + $USE_MVND = $true + $distributionUrl = $distributionUrl -replace '-bin\.[^.]*$',"-windows-amd64.zip" + $MVN_CMD = "mvnd.cmd" + break + } + default { + $USE_MVND = $false + $MVN_CMD = $script -replace '^mvnw','mvn' + break + } +} + +# apply MVNW_REPOURL and calculate MAVEN_HOME +# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-,maven-mvnd--}/ +if ($env:MVNW_REPOURL) { + $MVNW_REPO_PATTERN = if ($USE_MVND) { "/org/apache/maven/" } else { "/maven/mvnd/" } + $distributionUrl = "$env:MVNW_REPOURL$MVNW_REPO_PATTERN$($distributionUrl -replace '^.*'+$MVNW_REPO_PATTERN,'')" +} +$distributionUrlName = $distributionUrl -replace '^.*/','' +$distributionUrlNameMain = $distributionUrlName -replace '\.[^.]*$','' -replace '-bin$','' +$MAVEN_HOME_PARENT = "$HOME/.m2/wrapper/dists/$distributionUrlNameMain" +if ($env:MAVEN_USER_HOME) { + $MAVEN_HOME_PARENT = "$env:MAVEN_USER_HOME/wrapper/dists/$distributionUrlNameMain" +} +$MAVEN_HOME_NAME = ([System.Security.Cryptography.MD5]::Create().ComputeHash([byte[]][char[]]$distributionUrl) | ForEach-Object {$_.ToString("x2")}) -join '' +$MAVEN_HOME = "$MAVEN_HOME_PARENT/$MAVEN_HOME_NAME" + +if (Test-Path -Path "$MAVEN_HOME" -PathType Container) { + Write-Verbose "found existing MAVEN_HOME at $MAVEN_HOME" + Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD" + exit $? +} + +if (! $distributionUrlNameMain -or ($distributionUrlName -eq $distributionUrlNameMain)) { + Write-Error "distributionUrl is not valid, must end with *-bin.zip, but found $distributionUrl" +} + +# prepare tmp dir +$TMP_DOWNLOAD_DIR_HOLDER = New-TemporaryFile +$TMP_DOWNLOAD_DIR = New-Item -Itemtype Directory -Path "$TMP_DOWNLOAD_DIR_HOLDER.dir" +$TMP_DOWNLOAD_DIR_HOLDER.Delete() | Out-Null +trap { + if ($TMP_DOWNLOAD_DIR.Exists) { + try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null } + catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" } + } +} + +New-Item -Itemtype Directory -Path "$MAVEN_HOME_PARENT" -Force | Out-Null + +# Download and Install Apache Maven +Write-Verbose "Couldn't find MAVEN_HOME, downloading and installing it ..." +Write-Verbose "Downloading from: $distributionUrl" +Write-Verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName" + +$webclient = New-Object System.Net.WebClient +if ($env:MVNW_USERNAME -and $env:MVNW_PASSWORD) { + $webclient.Credentials = New-Object System.Net.NetworkCredential($env:MVNW_USERNAME, $env:MVNW_PASSWORD) +} +[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 +$webclient.DownloadFile($distributionUrl, "$TMP_DOWNLOAD_DIR/$distributionUrlName") | Out-Null + +# If specified, validate the SHA-256 sum of the Maven distribution zip file +$distributionSha256Sum = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionSha256Sum +if ($distributionSha256Sum) { + if ($USE_MVND) { + Write-Error "Checksum validation is not supported for maven-mvnd. `nPlease disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." + } + Import-Module $PSHOME\Modules\Microsoft.PowerShell.Utility -Function Get-FileHash + if ((Get-FileHash "$TMP_DOWNLOAD_DIR/$distributionUrlName" -Algorithm SHA256).Hash.ToLower() -ne $distributionSha256Sum) { + Write-Error "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised. If you updated your Maven version, you need to update the specified distributionSha256Sum property." + } +} + +# unzip and move +Expand-Archive "$TMP_DOWNLOAD_DIR/$distributionUrlName" -DestinationPath "$TMP_DOWNLOAD_DIR" | Out-Null +Rename-Item -Path "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain" -NewName $MAVEN_HOME_NAME | Out-Null +try { + Move-Item -Path "$TMP_DOWNLOAD_DIR/$MAVEN_HOME_NAME" -Destination $MAVEN_HOME_PARENT | Out-Null +} catch { + if (! (Test-Path -Path "$MAVEN_HOME" -PathType Container)) { + Write-Error "fail to move MAVEN_HOME" + } +} finally { + try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null } + catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" } +} + +Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD" diff --git a/pom.xml b/pom.xml new file mode 100644 index 000000000..febc5e3cd --- /dev/null +++ b/pom.xml @@ -0,0 +1,161 @@ + + + 4.0.0 + + org.springframework.boot + spring-boot-starter-parent + 3.3.3 + + + com.example.yuga.start1 + start + 0.0.1-SNAPSHOT + start + Demo project for Spring Boot + + + + + + + + + + + + + + + 17 + + + + org.springframework.boot + spring-boot-starter-data-jpa + + + org.springframework.boot + spring-boot-starter-thymeleaf + + + org.springframework.boot + spring-boot-starter-web + + + org.springframework.boot + spring-boot-starter-webflux + + + javax.annotation + javax.annotation-api + 1.3.2 + + + + org.springframework.boot + spring-boot-devtools + runtime + true + + + + + + + org.postgresql + postgresql + LATEST + + + org.json + json + LATEST + + + + + org.springframework.boot + spring-boot-starter-test + test + + + io.projectreactor + reactor-test + test + + + + org.springframework.data + spring-data-jpa + + + + com.yugabyte + jdbc-yugabytedb + 42.3.5-yb-5 + + + jakarta.persistence + jakarta.persistence-api + + + + + + com.google.api-client + google-api-client + 1.31.1 + + + com.google.guava + guava-jdk5 + + + + + + com.google.api-client + google-api-client-jackson2 + 1.31.1 + + + + com.google.oauth-client + google-oauth-client-jetty + 1.34.1 + + + com.google.apis + google-api-services-sheets + v4-rev612-1.25.0 + + + com.google.apis + google-api-services-drive + v3-rev20240123-2.0.0 + + + com.google.auth + google-auth-library-oauth2-http + 1.25.0 + + + + com.google.guava + guava + 33.3.0-jre + + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + + + diff --git a/src/main/java/com/example/yuga/start/StartApplication.java b/src/main/java/com/example/yuga/start/StartApplication.java new file mode 100644 index 000000000..b6a94f771 --- /dev/null +++ b/src/main/java/com/example/yuga/start/StartApplication.java @@ -0,0 +1,77 @@ +package com.example.yuga.start; + + +import com.example.yuga.start.gsheet.SheetsService; +import com.example.yuga.start.service.DynamicTableService; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.CommandLineRunner; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import java.io.IOException; +import java.util.*; + +@SpringBootApplication +public class StartApplication implements CommandLineRunner { + + @Autowired + DynamicTableService dynamicTableService; + @Autowired + private SheetsService gglsheetservice; +// @Autowired +// private GoogleSheetsConfig googleSheetsConfig; + + public static void main(String[] args) { + SpringApplication.run(StartApplication.class, args); + } + + @Override + public void run(String... args) throws Exception { + // Example table name and columns + Scanner in = new Scanner(System.in); +// String tableName = "students"; +// Map columns = Map.of( +// "srn", "TEXT PRIMARY KEY", +// "name", "VARCHAR(255)", +// "mail", "VARCHAR(255)" +// ); + System.out.println("want to create new table?"); + if(in.next().equals("yes")) { + System.out.println("enter sheet id"); + String sheetid = in.next(); + System.out.println("enter table name"); + String tableName = in.next(); + System.out.println("enter columns"); + HashMap columns = new LinkedHashMap<>(); + System.out.println("enter primary key"); + String pkey = in.next(); + columns.put(pkey, "TEXT PRIMARY KEY"); + System.out.println("enter other columns"); + String column = in.next(); + while (!column.equals("exit")) { + columns.put(column, "VARCHAR(255)"); + column = in.next(); + } + // Create table dynamically + dynamicTableService.createTable(sheetid,tableName, columns); + System.out.println("Created in sql table: " + tableName); + + gglsheetservice.addTableToSheet(sheetid, tableName, new ArrayList<>(columns.keySet())); + System.out.println("Created in gsheets table: " + tableName); + } +// System.out.println("enter sheet id"); +// String id = in.next(); + // Insert data dynamically +// Map rowData = Map.of( +// "srn", "PES1UG21CS247", +// "name", "John Doe", +// "mail", "john.doe@example.com" +// ); +// dynamicTableService.insertData(tableName, rowData); +// System.out.println("Inserted data into table: " + tableName); +//// +// // Query and print data dynamically +// var result = dynamicTableService.getAllData(tableName); +// System.out.println("Data from table: " + result); + + } +} diff --git a/src/main/java/com/example/yuga/start/controller/NotificationController.java b/src/main/java/com/example/yuga/start/controller/NotificationController.java new file mode 100644 index 000000000..9fef6d8e9 --- /dev/null +++ b/src/main/java/com/example/yuga/start/controller/NotificationController.java @@ -0,0 +1,31 @@ +package com.example.yuga.start.controller; + +import com.example.yuga.start.service.DynamicTableService; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.web.bind.annotation.*; + +@RestController +@RequestMapping() +@CrossOrigin(origins = "*",allowedHeaders = "*") +public class NotificationController { + + @Autowired + DynamicTableService dynamicTableService; +// @GetMapping("/notification") +// public void handleNotification() { +// // Process the payload +// System.out.println("Received notification: " ); +// +// // Example: Log the notification data +// // Log or handle the notification data as needed +// } + @PostMapping("/notifications") + public void handleNotifications(@RequestBody NotificationPayload payload) { + // Process the payload + System.out.println("Received notifications: " + payload); + dynamicTableService.scanpayload(payload); + + // Example: Log the notification data + // Log or handle the notification data as needed + } +} diff --git a/src/main/java/com/example/yuga/start/controller/NotificationPayload.java b/src/main/java/com/example/yuga/start/controller/NotificationPayload.java new file mode 100644 index 000000000..88e58192c --- /dev/null +++ b/src/main/java/com/example/yuga/start/controller/NotificationPayload.java @@ -0,0 +1,76 @@ +package com.example.yuga.start.controller; + +public class NotificationPayload { + private String sheetid; + private String columnName; + private String oldValue; + private String newValue; + private String id; // Use Integer to handle null values + private String timestamp; + + // Getters and Setters + + + public NotificationPayload() { + } + + public String getSheetid() { + return sheetid; + } + + public void setSheetid(String sheetid) { + this.sheetid = sheetid; + } + + public String getColumnName() { + return columnName; + } + + public void setColumnName(String columnName) { + this.columnName = columnName; + } + + public String getOldValue() { + return oldValue; + } + + public void setOldValue(String oldValue) { + this.oldValue = oldValue; + } + + public String getNewValue() { + return newValue; + } + + public void setNewValue(String newValue) { + this.newValue = newValue; + } + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public String getTimestamp() { + return timestamp; + } + + public void setTimestamp(String timestamp) { + this.timestamp = timestamp; + } + + @Override + public String toString() { + return "NotificationPayload{" + + "sheetid='" + sheetid + '\'' + + ", columnName='" + columnName + '\'' + + ", oldValue='" + oldValue + '\'' + + ", newValue='" + newValue + '\'' + + ", id=" + id + + ", timestamp='" + timestamp + '\'' + + '}'; + } +} diff --git a/src/main/java/com/example/yuga/start/gsheet/GoogleSheetsConfig.java b/src/main/java/com/example/yuga/start/gsheet/GoogleSheetsConfig.java new file mode 100644 index 000000000..bfad67e0e --- /dev/null +++ b/src/main/java/com/example/yuga/start/gsheet/GoogleSheetsConfig.java @@ -0,0 +1,51 @@ +package com.example.yuga.start.gsheet; + +import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport; +import com.google.api.client.http.HttpRequestInitializer; +import com.google.api.client.json.jackson2.JacksonFactory; +import com.google.api.services.drive.Drive; +import com.google.api.services.drive.DriveScopes; +import com.google.api.services.sheets.v4.Sheets; +import com.google.api.services.sheets.v4.SheetsScopes; +import com.google.auth.http.HttpCredentialsAdapter; +import com.google.auth.oauth2.GoogleCredentials; +import com.google.auth.oauth2.ServiceAccountCredentials; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +import java.io.FileInputStream; +import java.io.IOException; +import java.security.GeneralSecurityException; +import java.util.Arrays; +import java.util.List; + +@Configuration +public class GoogleSheetsConfig { + + private static final String APPLICATION_NAME = "gsheetmap"; + private static final String CREDENTIALS_FILE_PATH = "credentials.json"; + private static final List SCOPES = Arrays.asList(SheetsScopes.SPREADSHEETS, DriveScopes.DRIVE_FILE); + + private GoogleCredentials getCredentials() throws IOException { + return ServiceAccountCredentials.fromStream(new FileInputStream(CREDENTIALS_FILE_PATH)) + .createScoped(SCOPES); + } + + @Bean + public Sheets googleSheetsService() throws IOException, GeneralSecurityException { + HttpRequestInitializer requestInitializer = new HttpCredentialsAdapter(getCredentials()); + + return new Sheets.Builder(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(), requestInitializer) + .setApplicationName(APPLICATION_NAME) + .build(); + } + + @Bean + public Drive googleDriveService() throws IOException, GeneralSecurityException { + HttpRequestInitializer requestInitializer = new HttpCredentialsAdapter(getCredentials()); + + return new Drive.Builder(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(), requestInitializer) + .setApplicationName(APPLICATION_NAME) + .build(); + } +} \ No newline at end of file diff --git a/src/main/java/com/example/yuga/start/gsheet/SheetsService.java b/src/main/java/com/example/yuga/start/gsheet/SheetsService.java new file mode 100644 index 000000000..fef6e380a --- /dev/null +++ b/src/main/java/com/example/yuga/start/gsheet/SheetsService.java @@ -0,0 +1,242 @@ +package com.example.yuga.start.gsheet; + +import com.google.api.services.drive.Drive; +import com.google.api.services.drive.model.Permission; +import com.google.api.services.sheets.v4.Sheets; +import com.google.api.services.sheets.v4.model.*; +import org.springframework.stereotype.Service; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +//import java.security.Permission; + +@Service +public class SheetsService { + + private final Sheets sheetsService; + private final Drive driveService; + + public SheetsService(Sheets sheetsService, Drive driveService) { + this.sheetsService = sheetsService; + this.driveService = driveService; + } + + private static final String SPREADSHEETS_URL = "https://sheets.googleapis.com/v4/spreadsheets"; + + public Spreadsheet createSpreadsheet(String title) throws IOException { + System.out.println("Creating spreadsheet: " + title); + + // Create a new spreadsheet + Spreadsheet spreadsheet = new Spreadsheet() + .setProperties(new SpreadsheetProperties().setTitle(title)); + + // Request to create the spreadsheet + spreadsheet = sheetsService.spreadsheets().create(spreadsheet).execute(); + + System.out.println("Spreadsheet created with ID: " + spreadsheet.getSpreadsheetId()); + + // Make the spreadsheet public + try { + makePublic(spreadsheet.getSpreadsheetId()); + } catch (Exception e) { + System.err.println("Error making spreadsheet public: " + e.getMessage()); + e.printStackTrace(); + } + + return spreadsheet; + } + + public void makePublic(String fileId) throws IOException { + System.out.println("Attempting to make spreadsheet public. File ID: " + fileId); + + Permission newPermission = new Permission() + .setType("anyone") + .setRole("writer"); + + try { + Permission result = driveService.permissions().create(fileId, newPermission) + .setFields("id") + .setTransferOwnership(true) + .execute(); + System.out.println("Spreadsheet made public. Permission ID: " + result.getId()); + } catch (Exception e) { + System.err.println("Error setting permission: " + e.getMessage()); + e.printStackTrace(); + } + } + + public void addTableToSheet(String spreadsheetId, String sheetName, List columnNames) throws IOException { + // Define the range where the column headers will be placed + String range = "!A1:" + getColumnLetter(columnNames.size()) + "1"; + + // Prepare the values to be written + List> values = new ArrayList<>(); + List row = new ArrayList<>(); // Create a row list + + // Add each column name as a separate cell in the row + for (String columnName : columnNames) { + row.add(columnName); + } + + values.add(row); + System.out.println("values size"+values.size());// Add column names to the first row + System.out.println("values"+ values); + + ValueRange body = new ValueRange().setValues(values); + + UpdateValuesResponse result = sheetsService.spreadsheets().values() + .update(spreadsheetId, range, body) + .setValueInputOption("USER_ENTERED") // Use "RAW" to input data as-is or "USER_ENTERED" for user-friendly formatting + .execute(); + + System.out.println("Updated cells: " + result.getUpdatedCells()); + } + + public void insertData(String spreadsheetId, Map data) throws IOException { + // Fetch the existing data to search for the row where column A matches the value in the map + String range = "!A:A"; // We're only interested in column A for finding the correct row + ValueRange response = sheetsService.spreadsheets().values() + .get(spreadsheetId, range) + .execute(); + List> existingData = response.getValues(); + + int targetRow = -1; // Initialize target row to indicate not found + String columnAValue = data.get("A").toString(); // Get the value for column A from the map + + // Search for the row where column A matches the value provided + if (existingData != null) { + for (int i = 0; i < existingData.size(); i++) { + if (existingData.get(i).size() > 0 && columnAValue.equals(existingData.get(i).get(0).toString())) { + targetRow = i + 1; // Google Sheets rows are 1-based, so we add 1 + break; + } + } + } + + // If the row is not found, append data after the last filled row + if (targetRow == -1) { + targetRow = existingData.size() + 1; // Set target row to the next empty row + } + + // Prepare the data to be inserted + List> rowData = new ArrayList<>(); + List row = new ArrayList<>(); + + // Determine the range dynamically based on the map keys (columns) + List columns = new ArrayList<>(data.keySet()); + columns.sort(String::compareTo); // Sort keys to maintain column order (e.g., A, B, C) + + for (String column : columns) { + row.add(data.get(column)); // Add the corresponding value to the row + } + + rowData.add(row); + + // Define the range where the data will be placed based on the first and last columns in the map + String startColumn = columns.get(0); // First column in the map + String endColumn = columns.get(columns.size() - 1); // Last column in the map + String updateRange = "!" + startColumn + targetRow + ":" + endColumn + targetRow; + + // Prepare the values to be written + ValueRange body = new ValueRange().setValues(rowData); + UpdateValuesResponse result = sheetsService.spreadsheets().values() + .update(spreadsheetId, updateRange, body) + .setValueInputOption("USER_ENTERED") // Use "RAW" or "USER_ENTERED" based on your needs + .execute(); + + System.out.println("Updated cells: " + result.getUpdatedCells()); + } + + public void deleteRowIfExists(String spreadsheetId, Map data) throws IOException { + // Fetch the existing data to search for the row where column A matches the value in the map + String range = "!A:A"; // We're only interested in column A for finding the correct row + ValueRange response = sheetsService.spreadsheets().values() + .get(spreadsheetId, range) + .execute(); + List> existingData = response.getValues(); + + int targetRow = -1; // Initialize target row to indicate not found + String columnAValue = data.get("A").toString(); // Get the value for column A (the unique ID) + + // Search for the row where column A matches the value provided (unique ID) + if (existingData != null) { + for (int i = 0; i < existingData.size(); i++) { + if (existingData.get(i).size() > 0 && columnAValue.equals(existingData.get(i).get(0).toString())) { + targetRow = i + 1; // Google Sheets rows are 1-based, so we add 1 + break; + } + } + } + + // If the row is found, delete the row + if (targetRow != -1) { + deleteRow(spreadsheetId, targetRow); + } else { + System.out.println("Row with ID " + columnAValue + " not found. No row deleted."); + } + } + + private void deleteRow(String spreadsheetId, int rowIndex) throws IOException { + // Create a request to delete the row at the specified index + BatchUpdateSpreadsheetRequest batchUpdateRequest = new BatchUpdateSpreadsheetRequest() + .setRequests(Collections.singletonList( + new Request().setDeleteDimension(new DeleteDimensionRequest() + .setRange(new DimensionRange() + .setSheetId(getSheetId(spreadsheetId)) + .setDimension("ROWS") + .setStartIndex(rowIndex - 1) // 0-based index + .setEndIndex(rowIndex) // Exclusive end index + ) + ) + )); + + // Execute the batch update request + sheetsService.spreadsheets().batchUpdate(spreadsheetId, batchUpdateRequest).execute(); + System.out.println("Deleted row: " + rowIndex); + } + + // Helper method to get sheet ID based on the spreadsheet ID + private int getSheetId(String spreadsheetId) throws IOException { + Spreadsheet spreadsheet = sheetsService.spreadsheets().get(spreadsheetId).execute(); + return spreadsheet.getSheets().get(0).getProperties().getSheetId(); // Assumes the first sheet; adapt as needed + } + + + private String getColumnLetter(int columnIndex) { + StringBuilder column = new StringBuilder(); + while (columnIndex > 0) { + int modulo = (columnIndex - 1) % 26; + column.insert(0, (char) (65 + modulo)); + columnIndex = (columnIndex - modulo) / 26; + } + return column.toString(); + } + + public void readAllContentFromSheet(String spreadsheetId, String sheetName) throws IOException { + // Define the range for the entire sheet + String range = "A3:C3"; // This reads all data in the specified sheet + + // Call the Sheets API to get the values + ValueRange response = sheetsService.spreadsheets().values() + .get(spreadsheetId, range) + .execute(); + + // Get the values from the response + List> values = response.getValues(); + + // Check if there are any values + if (values == null || values.isEmpty()) { + System.out.println("No data found."); + } else { + // Print the values + for (List row : values) { + System.out.println(row); + } + } + + + } +} diff --git a/src/main/java/com/example/yuga/start/repos/DynamicTableRepository.java b/src/main/java/com/example/yuga/start/repos/DynamicTableRepository.java new file mode 100644 index 000000000..e85cf14cc --- /dev/null +++ b/src/main/java/com/example/yuga/start/repos/DynamicTableRepository.java @@ -0,0 +1,59 @@ +package com.example.yuga.start.repos; + +import com.example.yuga.start.controller.NotificationPayload; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.stereotype.Repository; +import java.util.List; +import java.util.Map; + +@Repository +public class DynamicTableRepository { + + private final JdbcTemplate jdbcTemplate; + + public DynamicTableRepository(JdbcTemplate jdbcTemplate) { + this.jdbcTemplate = jdbcTemplate; + } + + // Fetch all records dynamically + public List> findAll(String tableName) { + String query = String.format("SELECT * FROM %s", tableName); + return jdbcTemplate.queryForList(query); + } + + // Insert data dynamically + public void insertrow(String tableName, String column, String value) { + // Properly format the SQL query + String query = String.format("INSERT INTO %s (%s) VALUES (?)", tableName, column); + + // Execute the query with the value parameter + jdbcTemplate.update(query, value); + } + + // Update data dynamically + public void updateData(String tableName, String key, String val, String pkey, String pkval) { + // Construct the SQL query with the table name and column names directly + String query = String.format("UPDATE %s SET %s = ? WHERE %s = ?", tableName, key, pkey); + + // Execute the update operation with the actual values for placeholders + jdbcTemplate.update(query, val, pkval); + } + + + // Delete data dynamically + public void deleteData(String tableName, String pkey, String pkeyval) { + String query = String.format("DELETE FROM %s WHERE %s = ?", tableName, pkey); + + System.out.println("Executing query: " + query); + System.out.println("With value: " + pkeyval); + + // Ensure `pkeyval` is properly handled if it's a string + int rowsAffected = jdbcTemplate.update(query, pkeyval); + + System.out.println("Rows affected: " + rowsAffected); + } + + + + +} diff --git a/src/main/java/com/example/yuga/start/repos/SchemaRegistryRepository.java b/src/main/java/com/example/yuga/start/repos/SchemaRegistryRepository.java new file mode 100644 index 000000000..aeb529541 --- /dev/null +++ b/src/main/java/com/example/yuga/start/repos/SchemaRegistryRepository.java @@ -0,0 +1,129 @@ +package com.example.yuga.start.repos; + +import com.example.yuga.start.service.SqlNotificationPayload; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.stereotype.Repository; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +@Repository +public class SchemaRegistryRepository { + + private final JdbcTemplate jdbcTemplate; + + public SchemaRegistryRepository(JdbcTemplate jdbcTemplate) { + this.jdbcTemplate = jdbcTemplate; + } + + + public boolean tableExists(String tableName) { + String query = "SELECT COUNT(*) FROM schema_registry WHERE table_name = ?"; + int count = jdbcTemplate.queryForObject(query, new Object[]{tableName}, Integer.class); + System.out.println("count="+count); + return count > 0; + } + + public String gsheetcolName(String tableName,String sqlcolName){ + String query = "SELECT gsheetcol FROM schema_registry WHERE table_name = ? and column_name = ?"; + String gsheetcol = jdbcTemplate.queryForObject(query, new Object[]{tableName,sqlcolName},String.class ); + return gsheetcol; + } + + public List getColumnsForTable(String tableName) { + String query = "SELECT column_name FROM schema_registry WHERE table_name = ?"; + return jdbcTemplate.queryForList(query, new Object[]{tableName}, String.class); + } + + public void insertSchema(String tableName, Map columns){ + int i =0; + for (Map.Entry column : columns.entrySet()) { + String gsheetcol = getGoogleSheetColumn(i++); + String insertMetadataSql = "INSERT INTO schema_registry (table_name, column_name, column_type,gsheetcol) VALUES (?, ?, ?,?)"; + jdbcTemplate.update(insertMetadataSql, tableName, column.getKey(), column.getValue(),gsheetcol); + } + + System.out.println("Table created and schema registered: " + tableName); + } + public String getGoogleSheetColumn(int index) { + StringBuilder column = new StringBuilder(); + while (index >= 0) { + column.append((char) ('A' + (index % 26))); + index = (index / 26) - 1; + } + return column.reverse().toString(); // Return column letter only, without row number + } + + public Map getGSheetColumnMapping(SqlNotificationPayload payload) { + // Extract the table name + String tableName = extractTableName(payload); + System.out.println("Extracted table name: " + tableName); + + // Query to get the column mappings + String query = "SELECT column_name, gsheetcol " + + "FROM schema_registry " + + "WHERE table_name = ? AND gsheetcol IS NOT NULL"; + + // Execute the query and build the mapping + Map gsheetMapping = new HashMap<>(); + jdbcTemplate.query(query, new Object[]{tableName}, (rs) -> { + String columnName = rs.getString("column_name"); + String gsheetCol = rs.getString("gsheetcol"); + String value = payload.getData().get(columnName); + System.out.println("Mapping found: column=" + columnName + ", gsheetCol=" + gsheetCol + ", value=" + value); + if (value != null) { + gsheetMapping.put(gsheetCol, value); + } + }); + + System.out.println("Final GSheet mapping: " + gsheetMapping); + return gsheetMapping; + } + + private String extractTableName(SqlNotificationPayload payload) { + // First, try to extract from the type field + String[] typeParts = payload.getType().split(" ", 2); + if (typeParts.length > 1) { + return typeParts[1].toLowerCase().trim(); + } + + // If that fails, use a default table name or another method to determine it + return ""; // Default to 'boys' if we can't extract it from the type + } + + public void insertIntoSchemaLink(String link, String tableName) { + String query = "INSERT INTO schema_link (link, table_name) VALUES (?, ?)"; + + jdbcTemplate.update(query, link, tableName); + System.out.println("updates link"); + } + + public List getTableandprimarykeyfomschemalink(String link){ + String query = "SELECT table_name FROM schema_link WHERE link = ?"; + String table_name = jdbcTemplate.queryForObject(query, new Object[]{link},String.class ); + System.out.println("table is "+table_name); + List details = new ArrayList<>(); + details.add(table_name); + query = "SELECT column_name " + + "FROM schema_registry " + + "WHERE table_name = ? " + + "AND column_type LIKE '%PRIMARY KEY%'"; + + String pkey = jdbcTemplate.queryForObject(query, new Object[]{table_name},String.class ); + details.add(pkey); + return details; + + } + + public String getLink(String table_name){ + String query = "SELECT link FROM schema_link WHERE table_name = ?"; + String link = jdbcTemplate.queryForObject(query, new Object[]{table_name},String.class ); + return link; + } + + + + +} diff --git a/src/main/java/com/example/yuga/start/security/SimpleCORSFilter.java b/src/main/java/com/example/yuga/start/security/SimpleCORSFilter.java new file mode 100644 index 000000000..26f9bb474 --- /dev/null +++ b/src/main/java/com/example/yuga/start/security/SimpleCORSFilter.java @@ -0,0 +1,34 @@ +package com.example.yuga.start.security; + + +import jakarta.servlet.*; +import jakarta.servlet.http.HttpServletRequest; +import jakarta.servlet.http.HttpServletResponse; +import org.springframework.core.annotation.Order; +import org.springframework.stereotype.Component; + +import java.io.IOException; + +@Order(-1) +@Component +public class SimpleCORSFilter implements Filter { + + @Override + public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) throws IOException, ServletException { + + HttpServletRequest request = (HttpServletRequest) req; + HttpServletResponse response = (HttpServletResponse) res; + + System.out.println("started"); +// response.setHeader("Access-Control-Allow-Origin", request.getHeader("Origin")); + response.setHeader("Access-Control-Allow-Origin", "*"); // * = all domainName + response.setHeader("Access-Control-Allow-Credentials", "true"); // allow CrossDomain to use Origin Domain + response.setHeader("Access-Control-Allow-Methods", "POST, GET, OPTIONS, DELETE, PUT"); + response.setHeader("Access-Control-Max-Age", "3600"); // Preflight cache duration in browser +// response.setHeader("Access-Control-Allow-Headers", "Content-Type, Accept, X-Requested-With, remember-me"); + response.setHeader("Access-Control-Allow-Headers", "*"); // all header + + chain.doFilter(req, res); + } + +} \ No newline at end of file diff --git a/src/main/java/com/example/yuga/start/service/DynamicTableService.java b/src/main/java/com/example/yuga/start/service/DynamicTableService.java new file mode 100644 index 000000000..d432729ab --- /dev/null +++ b/src/main/java/com/example/yuga/start/service/DynamicTableService.java @@ -0,0 +1,214 @@ +package com.example.yuga.start.service; + +import com.example.yuga.start.controller.NotificationPayload; +import com.example.yuga.start.repos.DynamicTableRepository; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.stereotype.Service; + +import java.util.*; +import java.util.stream.Collectors; +import static java.lang.String.format; + +import java.util.Map; + +@Service +public class DynamicTableService { + + private final DynamicTableRepository dynamicTableRepository; + private final SchemaRegistryService schemaRegistryService; + private final JdbcTemplate jdbcTemplate; + + public DynamicTableService(DynamicTableRepository dynamicTableRepository, SchemaRegistryService schemaRegistryService, JdbcTemplate jdbcTemplate) { + this.dynamicTableRepository = dynamicTableRepository; + this.schemaRegistryService = schemaRegistryService; + this.jdbcTemplate = jdbcTemplate; + } + + public void createTable(String sheetid, String tableName, Map columns) { + // Validate that the table does not exist in the registry + if(schemaRegistryService.validateTableExists(tableName)){ + System.out.println("table"+tableName+"already exist"); + return; + } + + // Build SQL for creating the table + StringBuilder createTableSql = new StringBuilder("CREATE TABLE IF NOT EXISTS "); + createTableSql.append(tableName).append(" ("); + + // Append each column definition + columns.forEach((columnName, columnType) -> + createTableSql.append(columnName).append(" ").append(columnType).append(", ") + ); + + // Remove the last comma and space, then close the parenthesis + createTableSql.setLength(createTableSql.length() - 2); + createTableSql.append(")"); + System.out.println(createTableSql.toString()); + // Execute the SQL + jdbcTemplate.execute(createTableSql.toString()); + + System.out.println("Table created: " + tableName); + schemaRegistryService.addTable(tableName,columns); + createNotificationFunction(tableName, columns); + createNotificationTrigger(tableName); + schemaRegistryService.addschemalink(sheetid,tableName); + } + + + + public List> getAllData(String tableName) { + // Fetch schema details from the metadata service if necessary + schemaRegistryService.validateTableExists(tableName); + return dynamicTableRepository.findAll(tableName); + } + + public void insertrow(String tableName, String pkey,String pkeyval) { + // Validate that the table and column names match the metadata +// schemaRegistryService.validateColumns(tableName, data); + dynamicTableRepository.insertrow(tableName, pkey, pkeyval); + } + + public void updateData(String tableName, String key,String val, String pkey, String pkval) { + // Validate schema before update +// schemaRegistryService.validateColumns(tableName, data); + dynamicTableRepository.updateData(tableName, key,val,pkey,pkval); + } + + public void deleteData(String tableName, String pkey,String pkeyval) { +// String condition = "where %s = %s".formatted(pkey,pkeyval); + double number = Double.parseDouble(pkeyval); + pkeyval = Integer.toString((int)number); + System.out.println("gonna delete"); + dynamicTableRepository.deleteData(tableName, pkey,pkeyval); + } + + + public void createNotificationFunction(String tableName, Map columns) { + String functionName = "notify_" + tableName + "_changes"; + + List primaryKeys = getPrimaryKeyColumns(tableName); + + // Reverse the order of the column names (if needed) + List reversedColumns = new ArrayList<>(columns.keySet()); + // Collections.reverse(reversedColumns); // Uncomment if you want to reverse the order + + StringBuilder functionSql = new StringBuilder() + .append("CREATE OR REPLACE FUNCTION ").append(functionName) + .append("() RETURNS trigger AS $$\n") + .append("DECLARE\n") + .append(" operation TEXT;\n") + .append(" row_data JSONB;\n") + .append(" pk_data JSONB;\n") + .append("BEGIN\n") + .append(" IF TG_OP = 'INSERT' THEN\n") + .append(" operation := 'INSERT';\n") + .append(" row_data := jsonb_build_object(\n") + .append(reversedColumns.stream() + .map(col -> String.format(" '%s', NEW.%s", col, col)) + .collect(Collectors.joining(",\n"))) + .append("\n );\n") + .append(" pk_data := jsonb_build_object(\n") + .append(primaryKeys.stream() + .map(pk -> String.format(" '%s', NEW.%s", pk, pk)) + .collect(Collectors.joining(",\n"))) + .append("\n );\n") + .append(" ELSIF TG_OP = 'UPDATE' THEN\n") + .append(" operation := 'UPDATE';\n") + .append(" row_data := jsonb_build_object(\n") + .append(reversedColumns.stream() + .map(col -> String.format(" '%s', NEW.%s", col, col)) + .collect(Collectors.joining(",\n"))) + .append("\n );\n") + .append(" pk_data := jsonb_build_object(\n") + .append(primaryKeys.stream() + .map(pk -> String.format(" '%s', NEW.%s", pk, pk)) + .collect(Collectors.joining(",\n"))) + .append("\n );\n") + .append(" ELSIF TG_OP = 'DELETE' THEN\n") + .append(" operation := 'DELETE';\n") + .append(" row_data := jsonb_build_object(\n") + .append(reversedColumns.stream() + .map(col -> String.format(" '%s', OLD.%s", col, col)) + .collect(Collectors.joining(",\n"))) + .append("\n );\n") + .append(" pk_data := jsonb_build_object(\n") + .append(primaryKeys.stream() + .map(pk -> String.format(" '%s', OLD.%s", pk, pk)) + .collect(Collectors.joining(",\n"))) + .append("\n );\n") + .append(" END IF;\n") + .append(" PERFORM pg_notify('table_changes', \n") + .append(" format('Table: %s - Operation: %s - Data: %s - PrimaryKey: %s', \n") + .append(" '").append(tableName).append("',\n") + .append(" operation,\n") + .append(" row_data::text,\n") + .append(" pk_data::text\n") + .append(" )\n") + .append(" );\n") + .append(" RETURN NEW;\n") + .append("END;\n") + .append("$$ LANGUAGE plpgsql;\n"); + + System.out.println(functionSql.toString()); + // Execute the SQL to create the function + jdbcTemplate.execute(functionSql.toString()); + } + + + private List getPrimaryKeyColumns(String tableName) { + String query = "SELECT column_name " + + "FROM schema_registry " + + "WHERE table_name = ? " + + "AND column_type LIKE '%PRIMARY KEY%'"; + + return jdbcTemplate.query(query, new Object[]{tableName}, + (rs, rowNum) -> rs.getString("column_name")); + } + + + + + + private void createNotificationTrigger(String tableName) { + String functionName = "notify_" + tableName + "_changes"; + String triggerName = tableName + "_changes_trigger"; + StringBuilder triggerSql = new StringBuilder() + .append("CREATE TRIGGER ").append(triggerName) + .append("\nAFTER INSERT OR UPDATE OR DELETE ON ").append(tableName) + .append("\nFOR EACH ROW\n") + .append("EXECUTE FUNCTION ").append(functionName).append("();\n"); + + System.out.println(triggerSql.toString()); + + // Execute the SQL to create the trigger + jdbcTemplate.execute(triggerSql.toString()); + } + + public void scanpayload(NotificationPayload payload){ + List details = schemaRegistryService.getTablefomschemalink(payload.getSheetid()); + String pkey = details.get(1); + String table = details.get(0); + System.out.println("pkey size"+ payload.getId().length()); + if(payload.getColumnName().equals(pkey)){ + if(payload.getId()==null || payload.getId().isEmpty()){ + System.out.println("gonna delete"); + deleteData(table,pkey,payload.getOldValue()); + } + else { + insertrow(table,pkey, payload.getNewValue()); + } + } + else{ + String column_name = payload.getColumnName(); + String val = payload.getNewValue(); + String pkeyval = payload.getId(); + System.out.println("column name"+column_name); + System.out.println("val"+val); + System.out.println("pkey"+pkey); + System.out.println("pkeyval "+pkeyval); + updateData(table, column_name, val,pkey,pkeyval); + } + } + + +} diff --git a/src/main/java/com/example/yuga/start/service/PostgresNotificationService.java b/src/main/java/com/example/yuga/start/service/PostgresNotificationService.java new file mode 100644 index 000000000..535108738 --- /dev/null +++ b/src/main/java/com/example/yuga/start/service/PostgresNotificationService.java @@ -0,0 +1,87 @@ +package com.example.yuga.start.service; + +import com.example.yuga.start.gsheet.SheetsService; +import com.example.yuga.start.repos.SchemaRegistryRepository; +import org.postgresql.PGConnection; +import org.postgresql.PGNotification; +import org.postgresql.jdbc.PgConnection; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.stereotype.Service; + +import javax.annotation.PostConstruct; +import javax.annotation.PreDestroy; +import java.sql.Statement; +import java.util.Map; + +@Service +public class PostgresNotificationService { + + private final JdbcTemplate jdbcTemplate; + private final SchemaRegistryRepository schemaRegistryRepository; + private final SheetsService sheetsService; + private PgConnection pgConn; + + public PostgresNotificationService(JdbcTemplate jdbcTemplate, SchemaRegistryRepository schemaRegistryRepository, SheetsService sheetsService) { + this.jdbcTemplate = jdbcTemplate; + this.schemaRegistryRepository = schemaRegistryRepository; + this.sheetsService = sheetsService; + } + + @PostConstruct + public void startListener() throws Exception { + // Unwrap the existing connection to get PGConnection + pgConn = jdbcTemplate.getDataSource().getConnection().unwrap(PgConnection.class); + + // Create a statement to listen for notifications + Statement stmt = pgConn.createStatement(); + stmt.execute("LISTEN table_changes"); + + // Run the listener in a separate thread to avoid blocking the main thread + new Thread(this::listenForChanges).start(); + } + + private void listenForChanges() { + try { + System.out.println("Listening for notifications..."); + + while (true) { + // Check for notifications + PGNotification[] notifications = pgConn.getNotifications(); + + if (notifications != null) { + for (PGNotification notification : notifications) { + // Handle the notification +// System.out.println("Received notification: " + notification.getParameter().formatted()); + String ans = notification.getParameter(); + System.out.println("ans = "+ans); + SqlNotificationPayload sqlNotificationPayload = SqlNotificationPayload.fromNotification(ans); + // Further processing logic (e.g., trigger service or handle data) + System.out.println(sqlNotificationPayload.toString()); + + Map mapping =schemaRegistryRepository.getGSheetColumnMapping(sqlNotificationPayload); + if(sqlNotificationPayload.getType().equals("DELETE")){ + sheetsService.deleteRowIfExists("1dexgjeGyQt4ZBq8Z0fCaQRAIX7ARZKUkHMmNjRkgUTY",mapping); + } + else { + sheetsService.insertData("1dexgjeGyQt4ZBq8Z0fCaQRAIX7ARZKUkHMmNjRkgUTY", mapping); + } + + } + } + + // Sleep to reduce CPU usage + Thread.sleep(500); + } + } catch (Exception e) { + e.printStackTrace(); + } + } + + @PreDestroy + public void stopListener() throws Exception { + // Close the connection when shutting down the service + if (pgConn != null) { + pgConn.close(); + } + } +} diff --git a/src/main/java/com/example/yuga/start/service/SchemaRegistryService.java b/src/main/java/com/example/yuga/start/service/SchemaRegistryService.java new file mode 100644 index 000000000..092c195bd --- /dev/null +++ b/src/main/java/com/example/yuga/start/service/SchemaRegistryService.java @@ -0,0 +1,46 @@ +package com.example.yuga.start.service; + +import org.springframework.stereotype.Service; +import java.util.List; +import java.util.Map; +import com.example.yuga.start.repos.SchemaRegistryRepository; + +@Service +public class SchemaRegistryService { + + private final SchemaRegistryRepository schemaRegistryRepository; + + public SchemaRegistryService(SchemaRegistryRepository schemaRegistryRepository) { + this.schemaRegistryRepository = schemaRegistryRepository; + } + + public boolean validateTableExists(String tableName) { + // Check if the table exists in the registry + // throw new IllegalArgumentException("Table " + tableName + " does not exist"); + System.out.println("table existence ="+schemaRegistryRepository.tableExists(tableName)); + return schemaRegistryRepository.tableExists(tableName); + } + + public void validateColumns(String tableName, Map data) { + // Fetch columns from the registry + List validColumns = schemaRegistryRepository.getColumnsForTable(tableName); + System.out.println("the colums are:"+validColumns.toString()); + for (String column : data.keySet()) { + if (!validColumns.contains(column)) { + throw new IllegalArgumentException("Invalid column " + column + " for table " + tableName); + } + } + } + + public void addTable(String tableName, Map columns){ + schemaRegistryRepository.insertSchema(tableName,columns); + } + + public void addschemalink(String sheetid,String tablename){ + schemaRegistryRepository.insertIntoSchemaLink(sheetid,tablename); + } + + public List getTablefomschemalink(String sheetid){ + return schemaRegistryRepository.getTableandprimarykeyfomschemalink(sheetid); + } +} diff --git a/src/main/java/com/example/yuga/start/service/SqlNotificationPayload.java b/src/main/java/com/example/yuga/start/service/SqlNotificationPayload.java new file mode 100644 index 000000000..64d9c4773 --- /dev/null +++ b/src/main/java/com/example/yuga/start/service/SqlNotificationPayload.java @@ -0,0 +1,102 @@ +package com.example.yuga.start.service; + +import org.json.JSONObject; +import java.util.HashMap; + +public class SqlNotificationPayload { + private String type; + private HashMap data; + private String pkey; + private String tableName; + // Getters and setters + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public HashMap getData() { + return data; + } + + public void setData(HashMap data) { + this.data = data; + } + + public String getPkey() { + return pkey; + } + + public void setPkey(String pkey) { + this.pkey = pkey; + } + + public String getTableName() { + return tableName; + } + + public void setTableName(String tableName) { + this.tableName = tableName; + } + + // Method to parse notification and fill SqlNotificationPayload + public static SqlNotificationPayload fromNotification(String notification) { + SqlNotificationPayload payload = new SqlNotificationPayload(); + + String[] parts = notification.split(" - ", 4); // Split into 4 parts maximum + if (parts.length >= 3) { + // Extract table name + String[] tableParts = parts[0].split(":", 2); + if (tableParts.length == 2) { + payload.setTableName(tableParts[1].trim()); + } + + // Extract operation type + String[] operationParts = parts[1].split(":", 2); + if (operationParts.length == 2) { + payload.setType(operationParts[1].trim()); + } + + // Extract data and primary key + String dataAndPkPart = parts[parts.length - 1]; // Last part contains data and primary key + String[] dataParts = dataAndPkPart.split("PrimaryKey:", 2); + if (dataParts.length == 2) { + String dataJson = dataParts[0].replace("Data:", "").trim(); + String pkJson = dataParts[1].trim(); + + HashMap dataMap = new HashMap<>(); + + // Parse Data JSON + JSONObject dataObject = new JSONObject(dataJson); + dataObject.keys().forEachRemaining(key -> dataMap.put(key, dataObject.getString(key))); + + // Parse PrimaryKey JSON + JSONObject pkObject = new JSONObject(pkJson); + if (pkObject.length() > 0) { + String pkKey = pkObject.keys().next(); + String pkValue = pkObject.getString(pkKey); + payload.setPkey(pkValue); + // Also add the primary key to the data map if it's not already there + if (!dataMap.containsKey(pkKey)) { + dataMap.put(pkKey, pkValue); + } + } + + payload.setData(dataMap); + } + } + + return payload; + } + + @Override + public String toString() { + return "SqlNotificationPayload{" + + "type='" + type + '\'' + + ", data=" + data + + ", pkey='" + pkey + '\'' + + '}'; + } +} diff --git a/src/main/resources/application.properties b/src/main/resources/application.properties new file mode 100644 index 000000000..a2e51c39b --- /dev/null +++ b/src/main/resources/application.properties @@ -0,0 +1,26 @@ +spring.application.name=start +# Hibernate ddl auto (create, create-drop, validate, update, none). +spring.jpa.hibernate.ddl-auto=update +# --------------------- +# JPA/Hibernate config. +spring.jpa.database=POSTGRESQL + +# The SQL dialect makes Hibernate generate better SQL for the chosen database. +spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.PostgreSQLDialect + +# Hibernate ddl auto (create, create-drop, validate, update). +spring.jpa.show-sql=true +spring.jpa.generate-ddl=true +#spring.jpa.hibernate.ddl-auto=create + +# ------------------- +# Data-source config. +spring.sql.init.platform=postgres +spring.datasource.url=jdbc:postgresql://localhost:5432/super +spring.datasource.username=postgres +spring.datasource.password=JoeL@123! + +# HikariCP config (pool size, default isolation level). +spring.datasource.type=com.zaxxer.hikari.HikariDataSource +spring.datasource.hikari.transactionIsolation=TRANSACTION_SERIALIZABLE +#logging.level.org.springframework=DEBUG diff --git a/src/test/java/com/example/yuga/start/StartApplicationTests.java b/src/test/java/com/example/yuga/start/StartApplicationTests.java new file mode 100644 index 000000000..0ba836acc --- /dev/null +++ b/src/test/java/com/example/yuga/start/StartApplicationTests.java @@ -0,0 +1,13 @@ +package com.example.yuga.start; + +import org.junit.jupiter.api.Test; +import org.springframework.boot.test.context.SpringBootTest; + +@SpringBootTest +class StartApplicationTests { + + @Test + void contextLoads() { + } + +} diff --git a/src/test/java/com/example/yuga/start1/start/StartApplicationTests.java b/src/test/java/com/example/yuga/start1/start/StartApplicationTests.java new file mode 100644 index 000000000..189c4bec9 --- /dev/null +++ b/src/test/java/com/example/yuga/start1/start/StartApplicationTests.java @@ -0,0 +1,13 @@ +package com.example.yuga.start1.start; + +import org.junit.jupiter.api.Test; +import org.springframework.boot.test.context.SpringBootTest; + +@SpringBootTest +class StartApplicationTests { + + @Test + void contextLoads() { + } + +} From ccab5da6e66eb779378160b784db62869e49d911 Mon Sep 17 00:00:00 2001 From: Joel Renjith Date: Tue, 17 Sep 2024 00:52:45 +0530 Subject: [PATCH 18/23] d --- credentials.json | 4 +- .../start/StartApplication.java | 24 ++++- .../controller/NotificationController.java | 4 +- .../start/controller/NotificationPayload.java | 2 +- .../start/gsheet/GoogleSheetsConfig.java | 2 +- .../start/gsheet/SheetsService.java | 7 +- .../start/repos/DynamicTableRepository.java | 3 +- .../start/repos/SchemaRegistryRepository.java | 15 +-- .../start/security/SimpleCORSFilter.java | 2 +- .../start/service/DynamicTableService.java | 21 ++-- .../service/PostgresNotificationService.java | 20 ++-- .../start/service/SchemaRegistryService.java | 4 +- .../start/service/SqlNotificationPayload.java | 78 ++++++++++++++ .../start/service/SqlNotificationPayload.java | 102 ------------------ .../start/StartApplicationTests.java | 2 +- 15 files changed, 139 insertions(+), 151 deletions(-) rename src/main/java/com/example/{yuga => joel}/start/StartApplication.java (79%) rename src/main/java/com/example/{yuga => joel}/start/controller/NotificationController.java (90%) rename src/main/java/com/example/{yuga => joel}/start/controller/NotificationPayload.java (97%) rename src/main/java/com/example/{yuga => joel}/start/gsheet/GoogleSheetsConfig.java (98%) rename src/main/java/com/example/{yuga => joel}/start/gsheet/SheetsService.java (98%) rename src/main/java/com/example/{yuga => joel}/start/repos/DynamicTableRepository.java (94%) rename src/main/java/com/example/{yuga => joel}/start/repos/SchemaRegistryRepository.java (88%) rename src/main/java/com/example/{yuga => joel}/start/security/SimpleCORSFilter.java (97%) rename src/main/java/com/example/{yuga => joel}/start/service/DynamicTableService.java (93%) rename src/main/java/com/example/{yuga => joel}/start/service/PostgresNotificationService.java (81%) rename src/main/java/com/example/{yuga => joel}/start/service/SchemaRegistryService.java (94%) create mode 100644 src/main/java/com/example/joel/start/service/SqlNotificationPayload.java delete mode 100644 src/main/java/com/example/yuga/start/service/SqlNotificationPayload.java rename src/test/java/com/example/{yuga => joel}/start/StartApplicationTests.java (84%) diff --git a/credentials.json b/credentials.json index e140a8473..c5da59701 100644 --- a/credentials.json +++ b/credentials.json @@ -1,8 +1,8 @@ { "type": "service_account", "project_id": "gsheetssqlmap", - "private_key_id": "39798df6b3a4ffd23912f9a0e85590a145e39671", - "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCq8mNIvQQE8SM5\nuotOOr7RBjqpctGMlH4DRp5FcqpIz5FNla1AH3s5MIvEEo6L769Exdd+wtAnxdsF\nHaBZxHi58xRsyxkWFNPliQR8B4dqaQM1TxfJ8ocH0IUV/paSNa6uCLnucRMYw/Pu\nZb2kIocdmJSXBvl/BhHm6xNOvai8YOMHcN0fbN5sKCW+Wi3Vyd/kWLeYPn6FHj4T\nXuca6tBlbSALPDoVbcMwpdSfMZvjQ0jvxcZHkvOsfEKCeHxP46ylfkddmfaXywTT\nR0eEi64asFcexGGENE8RxqIVa68T3pWfPvxzWAr16KaVKzCutKIzBOFh636aB1/s\nep9AkSzpAgMBAAECggEAEwyq//FxQ9g3ydB2AuAUwMACFD0raRAomTaav0gzfeTk\njynjajJ/5av5XT6IpEMvy7kExjdRISeBHEk0g+BbKDSUdXmt+FoWkVzzExu3LUZZ\nwUxorRxRzA8Cb6Sk1JL3HeVyy8WNwFwgwBOWHhbraeLFhy1zquIrJ0mRp5wGtEQk\ng9JCY2RkTMH0Ro040UEgdo2oFo+61BMLSR9nnJYdP/0GZ6ooN+OJVEx+MhjDS37f\niJq7iN2MVd0I113SbRQGysmVz781yCoN7TcfBSO4qPX1Z1mMdm3wyv3K9e3afCgC\nS8aR8ZyG6YA1vW8GlLSVcA+pxrZ+GBRNR3hBaBCvUQKBgQDcpIv6s3MQZxbaCBoN\n93L4mmREQ3+/WyK+AqHcfvzrccw5OUkJDRfA3PN+7y39pYSoXGkJdu16HoFBQM8f\n6CgDEGt3iIDQ4VcNupWHBFvu8C8x/Pfk3N7LWqwewC0fzvGZhJbIu45shKdMyeqD\nYPzq8SShG0rfdxBfQpPKWGkhuQKBgQDGVyi5H5s5RhrTIvAz5wRkoSRQwro4TXTm\nV1sUpLqB3XFzpA7zAPl9rDwc1h+EB5cBzBmizCpXqBdUaCYh4e1CRwbdKJunolRK\njKLvANDKHAcRODeXNjll8I/ekKdFlQOu/5ZZBGlAbY/GRUyw5+8fIzxWgQ+ZVkuq\nhfq6loW8sQKBgQDCI+Kq/7Sx7RNVWyWZvm4iQq8QxZelBwqlx+kl+3KNtD6u5zHV\nTfjlkUSLHucHtNjpzw347vT7q9WtVdopx/tU3uDAd5JouK4BfXhOKc6+igXs5Ro1\n/QXoRmsGT7AkudEpdFm8XBNwdrRCQ7QeT3ubOf0x6LLTt47nUsx5kZtHuQKBgA5W\nBHKFD1IveAbk0pndy9p+L0LLIGyrZJS1oba3RWfXmriyFgaIfoAP7/RhobuIVOjj\nWMMEoNbfJniKYp82VlDhE2Tu53lRaWhODATceTrylawv9Qyv9awhnDSJHh4QbEi8\n3qAminkGskqxfZ6X1RfaEyVphLgfna6AqqPL93SxAoGAcatEfglIlVeHYkl4aUV4\nBZvRb35VJc3HTsXfeN0Bnk2Kue46cF3IEgKlXHz/cxhStFPiGRj2fz7LsdZzWPbZ\nmSjSXP/mPycMcYR55SMHhWI95BXDxaC6Eg8N9qP+9TbPm2c8si2KRYa2kbCYfETX\ntq4kjFWG/Z27jKWWi3+uttg=\n-----END PRIVATE KEY-----\n", + "private_key_id": "74fa4a5248c5447a3084ade43d3cf4469a575b2d", + "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDiK8F7/QvFjMFD\nIV5h0m6o2TjShSTT0HiDsNzN9qiIzijBAnnwg/525GxMvpK7yi0SqHYC3jvjR2FH\nRv6RighkayAeoZ2vwe7Gj0I0Rix5PHnXsw+/MmkshiKATyGLaDdf/Ihyb0K+Eaod\n2uecaCoWGFXiwYlD1MZzxRK/Nd4mMA1afQQcxr8l5YwtbuhlWuvKnUN5iJ1ievuF\nNJa2fKw9aYQD+7tL88V3rvLgtky3wVBNHbgWvO3piEg4E9oQzCmqF7fRuX+4Lggj\nxJAIoDHi0r1vsvfqFAL2dXPBAARvNwIjVlck5U134QlVoznx4LmvN9tspBVmxFgN\neY1H/jtPAgMBAAECggEARVjSOhCkD93xEgLEvCy4BmiYPiBZ4uJMWWCcy0Aie9Pu\ndgOq3N6z4sN2G2/hcNf0SwDBMm7uoOjtgFvZyUkv+o5IKmw2e8H0Uwhw7bNIhqHO\n4B7EqMStr9uu73WT3PlG5VT6h0/rAHlNFltc4b2kjeLHTsA1tMkkUjHuZhd/tmg4\nz+0F9nI82eODL0hYNII8OwBEoXXSj9687fIDwF96W45hoFFXftons8agVu4qHjdc\nZZsU28zZTXw8TJLrgN0Rql9VOUnDRNBBrVLzzcU0dccOkg9dbsCZknQCdv+IF2gg\nC/rcBfwWKtSyonhPttR3UM7szYMkUWa5hG3aZMvMLQKBgQD16TTeeOLui9r1orrb\nQ9UwA+BnLRz7MWqiA2PUw4cM2BlM0kJWRMzp0SiKAilzSOffbC14HZgO1SmqcZ1I\n/2q9wghGvoyXqGjDRWfJJE0ndhNCR3KxVbyRpt0HT72JXaaH5Ru7fOga+/fG/Mi4\n/eXgFoeCoLehm+C4VldroQQ+awKBgQDrczhqL+XWMWEVgmJErak3mQvj5rm9s2hI\nRjABFsZFsB/y0tNnw7fUdbIPpTiUz8LnHGBVzEp4vIjWuX9jVBDuWZ96CSsg6Xly\nvLvqiPeVySZJcEulZNqBbrYaHiHsS15T7uL6LL/IdUNA6wfeKXp4GGB5Jm2MAqaW\nqWoJg8lnrQKBgGUff1Q1FezOqj30qy5bmKFkFrWXAaHyUk1JY5Ysya5odw72L71W\nEYXdMvR9qxnzUxa3X4NTeCIxWi7zBaDtF7HsxoaA+UOsR1h1hBZczTV7ArRwuSNf\nhGnI2n+VNPX2AvNDLcJNvQzLrLqxXMFl90/NM/BtstXMSf7D0rM5xYr9AoGBAJ09\n/dcs1doOcD/lK6uiLviqhX5WgIVoY6Sg+z4+73tq/Gs20WjjFMoXMCuys+7uMcQW\n919VaAbqIpkEmUbExenogNEHfn4kbq9pzDX98xtdkuxIyrxSdnsqbQNUTGN6kiyL\nxZo+eHZvmdKpJEACg3/hB8zhFRgETS0hR2fDU6ahAoGBAMRVsxNkBSDJwb2vNSED\nTCFSLFdPQl4YzMhwonD/8fmr2yfroLGFD+VDCC5vu2a0XXyxfbRtMKAGC1eQXVIU\nHTV9dA7aUgyWKtb54QWZ1M419lF/7oQIMRE3B9eS+UhtRIaxVW9UFZvsuaCjUf5w\n2XpPJyzQ1+VbpRzneo03KaMV\n-----END PRIVATE KEY-----\n", "client_email": "gmapsheet@gsheetssqlmap.iam.gserviceaccount.com", "client_id": "108264969693417699931", "auth_uri": "https://accounts.google.com/o/oauth2/auth", diff --git a/src/main/java/com/example/yuga/start/StartApplication.java b/src/main/java/com/example/joel/start/StartApplication.java similarity index 79% rename from src/main/java/com/example/yuga/start/StartApplication.java rename to src/main/java/com/example/joel/start/StartApplication.java index b6a94f771..6562f5026 100644 --- a/src/main/java/com/example/yuga/start/StartApplication.java +++ b/src/main/java/com/example/joel/start/StartApplication.java @@ -1,13 +1,13 @@ -package com.example.yuga.start; +package com.example.joel.start; -import com.example.yuga.start.gsheet.SheetsService; -import com.example.yuga.start.service.DynamicTableService; +import com.example.joel.start.gsheet.SheetsService; +import com.example.joel.start.service.DynamicTableService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.CommandLineRunner; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; -import java.io.IOException; + import java.util.*; @SpringBootApplication @@ -37,7 +37,8 @@ public void run(String... args) throws Exception { System.out.println("want to create new table?"); if(in.next().equals("yes")) { System.out.println("enter sheet id"); - String sheetid = in.next(); + String url = in.next(); + String sheetid = extractSheetId(url); System.out.println("enter table name"); String tableName = in.next(); System.out.println("enter columns"); @@ -74,4 +75,17 @@ public void run(String... args) throws Exception { // System.out.println("Data from table: " + result); } + public static String extractSheetId(String url) { + // Regular expression to match the ID in the URL + String regex = "/d/([a-zA-Z0-9-_]+)"; + java.util.regex.Pattern pattern = java.util.regex.Pattern.compile(regex); + java.util.regex.Matcher matcher = pattern.matcher(url); + + if (matcher.find()) { + return matcher.group(1); + } else { + System.out.println("Invalid URL or ID not found"); + return null; + } + } } diff --git a/src/main/java/com/example/yuga/start/controller/NotificationController.java b/src/main/java/com/example/joel/start/controller/NotificationController.java similarity index 90% rename from src/main/java/com/example/yuga/start/controller/NotificationController.java rename to src/main/java/com/example/joel/start/controller/NotificationController.java index 9fef6d8e9..e53b1bd4e 100644 --- a/src/main/java/com/example/yuga/start/controller/NotificationController.java +++ b/src/main/java/com/example/joel/start/controller/NotificationController.java @@ -1,6 +1,6 @@ -package com.example.yuga.start.controller; +package com.example.joel.start.controller; -import com.example.yuga.start.service.DynamicTableService; +import com.example.joel.start.service.DynamicTableService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.*; diff --git a/src/main/java/com/example/yuga/start/controller/NotificationPayload.java b/src/main/java/com/example/joel/start/controller/NotificationPayload.java similarity index 97% rename from src/main/java/com/example/yuga/start/controller/NotificationPayload.java rename to src/main/java/com/example/joel/start/controller/NotificationPayload.java index 88e58192c..48186f313 100644 --- a/src/main/java/com/example/yuga/start/controller/NotificationPayload.java +++ b/src/main/java/com/example/joel/start/controller/NotificationPayload.java @@ -1,4 +1,4 @@ -package com.example.yuga.start.controller; +package com.example.joel.start.controller; public class NotificationPayload { private String sheetid; diff --git a/src/main/java/com/example/yuga/start/gsheet/GoogleSheetsConfig.java b/src/main/java/com/example/joel/start/gsheet/GoogleSheetsConfig.java similarity index 98% rename from src/main/java/com/example/yuga/start/gsheet/GoogleSheetsConfig.java rename to src/main/java/com/example/joel/start/gsheet/GoogleSheetsConfig.java index bfad67e0e..2e8579c6f 100644 --- a/src/main/java/com/example/yuga/start/gsheet/GoogleSheetsConfig.java +++ b/src/main/java/com/example/joel/start/gsheet/GoogleSheetsConfig.java @@ -1,4 +1,4 @@ -package com.example.yuga.start.gsheet; +package com.example.joel.start.gsheet; import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport; import com.google.api.client.http.HttpRequestInitializer; diff --git a/src/main/java/com/example/yuga/start/gsheet/SheetsService.java b/src/main/java/com/example/joel/start/gsheet/SheetsService.java similarity index 98% rename from src/main/java/com/example/yuga/start/gsheet/SheetsService.java rename to src/main/java/com/example/joel/start/gsheet/SheetsService.java index fef6e380a..254170e30 100644 --- a/src/main/java/com/example/yuga/start/gsheet/SheetsService.java +++ b/src/main/java/com/example/joel/start/gsheet/SheetsService.java @@ -1,4 +1,4 @@ -package com.example.yuga.start.gsheet; +package com.example.joel.start.gsheet; import com.google.api.services.drive.Drive; import com.google.api.services.drive.model.Permission; @@ -104,6 +104,7 @@ public void insertData(String spreadsheetId, Map data) throws IO List> existingData = response.getValues(); int targetRow = -1; // Initialize target row to indicate not found + try{ String columnAValue = data.get("A").toString(); // Get the value for column A from the map // Search for the row where column A matches the value provided @@ -115,6 +116,10 @@ public void insertData(String spreadsheetId, Map data) throws IO } } } + } + catch (Exception e){ +// System.out.println(""); + } // If the row is not found, append data after the last filled row if (targetRow == -1) { diff --git a/src/main/java/com/example/yuga/start/repos/DynamicTableRepository.java b/src/main/java/com/example/joel/start/repos/DynamicTableRepository.java similarity index 94% rename from src/main/java/com/example/yuga/start/repos/DynamicTableRepository.java rename to src/main/java/com/example/joel/start/repos/DynamicTableRepository.java index e85cf14cc..a08b60d83 100644 --- a/src/main/java/com/example/yuga/start/repos/DynamicTableRepository.java +++ b/src/main/java/com/example/joel/start/repos/DynamicTableRepository.java @@ -1,6 +1,5 @@ -package com.example.yuga.start.repos; +package com.example.joel.start.repos; -import com.example.yuga.start.controller.NotificationPayload; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.stereotype.Repository; import java.util.List; diff --git a/src/main/java/com/example/yuga/start/repos/SchemaRegistryRepository.java b/src/main/java/com/example/joel/start/repos/SchemaRegistryRepository.java similarity index 88% rename from src/main/java/com/example/yuga/start/repos/SchemaRegistryRepository.java rename to src/main/java/com/example/joel/start/repos/SchemaRegistryRepository.java index aeb529541..cb46efe37 100644 --- a/src/main/java/com/example/yuga/start/repos/SchemaRegistryRepository.java +++ b/src/main/java/com/example/joel/start/repos/SchemaRegistryRepository.java @@ -1,6 +1,6 @@ -package com.example.yuga.start.repos; +package com.example.joel.start.repos; -import com.example.yuga.start.service.SqlNotificationPayload; +import com.example.joel.start.service.SqlNotificationPayload; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.stereotype.Repository; @@ -58,7 +58,7 @@ public String getGoogleSheetColumn(int index) { public Map getGSheetColumnMapping(SqlNotificationPayload payload) { // Extract the table name - String tableName = extractTableName(payload); + String tableName = payload.getTableName(); System.out.println("Extracted table name: " + tableName); // Query to get the column mappings @@ -82,16 +82,7 @@ public Map getGSheetColumnMapping(SqlNotificationPayload payload return gsheetMapping; } - private String extractTableName(SqlNotificationPayload payload) { - // First, try to extract from the type field - String[] typeParts = payload.getType().split(" ", 2); - if (typeParts.length > 1) { - return typeParts[1].toLowerCase().trim(); - } - // If that fails, use a default table name or another method to determine it - return ""; // Default to 'boys' if we can't extract it from the type - } public void insertIntoSchemaLink(String link, String tableName) { String query = "INSERT INTO schema_link (link, table_name) VALUES (?, ?)"; diff --git a/src/main/java/com/example/yuga/start/security/SimpleCORSFilter.java b/src/main/java/com/example/joel/start/security/SimpleCORSFilter.java similarity index 97% rename from src/main/java/com/example/yuga/start/security/SimpleCORSFilter.java rename to src/main/java/com/example/joel/start/security/SimpleCORSFilter.java index 26f9bb474..f40bef175 100644 --- a/src/main/java/com/example/yuga/start/security/SimpleCORSFilter.java +++ b/src/main/java/com/example/joel/start/security/SimpleCORSFilter.java @@ -1,4 +1,4 @@ -package com.example.yuga.start.security; +package com.example.joel.start.security; import jakarta.servlet.*; diff --git a/src/main/java/com/example/yuga/start/service/DynamicTableService.java b/src/main/java/com/example/joel/start/service/DynamicTableService.java similarity index 93% rename from src/main/java/com/example/yuga/start/service/DynamicTableService.java rename to src/main/java/com/example/joel/start/service/DynamicTableService.java index d432729ab..51319c416 100644 --- a/src/main/java/com/example/yuga/start/service/DynamicTableService.java +++ b/src/main/java/com/example/joel/start/service/DynamicTableService.java @@ -1,7 +1,7 @@ -package com.example.yuga.start.service; +package com.example.joel.start.service; -import com.example.yuga.start.controller.NotificationPayload; -import com.example.yuga.start.repos.DynamicTableRepository; +import com.example.joel.start.controller.NotificationPayload; +import com.example.joel.start.repos.DynamicTableRepository; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.stereotype.Service; @@ -89,7 +89,7 @@ public void createNotificationFunction(String tableName, Map col List primaryKeys = getPrimaryKeyColumns(tableName); // Reverse the order of the column names (if needed) - List reversedColumns = new ArrayList<>(columns.keySet()); +// List reversedColumns = new ArrayList<>(columns.keySet()); // Collections.reverse(reversedColumns); // Uncomment if you want to reverse the order StringBuilder functionSql = new StringBuilder() @@ -103,7 +103,7 @@ public void createNotificationFunction(String tableName, Map col .append(" IF TG_OP = 'INSERT' THEN\n") .append(" operation := 'INSERT';\n") .append(" row_data := jsonb_build_object(\n") - .append(reversedColumns.stream() + .append(columns.keySet().stream() .map(col -> String.format(" '%s', NEW.%s", col, col)) .collect(Collectors.joining(",\n"))) .append("\n );\n") @@ -115,7 +115,7 @@ public void createNotificationFunction(String tableName, Map col .append(" ELSIF TG_OP = 'UPDATE' THEN\n") .append(" operation := 'UPDATE';\n") .append(" row_data := jsonb_build_object(\n") - .append(reversedColumns.stream() + .append(columns.keySet().stream() .map(col -> String.format(" '%s', NEW.%s", col, col)) .collect(Collectors.joining(",\n"))) .append("\n );\n") @@ -127,7 +127,7 @@ public void createNotificationFunction(String tableName, Map col .append(" ELSIF TG_OP = 'DELETE' THEN\n") .append(" operation := 'DELETE';\n") .append(" row_data := jsonb_build_object(\n") - .append(reversedColumns.stream() + .append(columns.keySet().stream() .map(col -> String.format(" '%s', OLD.%s", col, col)) .collect(Collectors.joining(",\n"))) .append("\n );\n") @@ -141,15 +141,18 @@ public void createNotificationFunction(String tableName, Map col .append(" format('Table: %s - Operation: %s - Data: %s - PrimaryKey: %s', \n") .append(" '").append(tableName).append("',\n") .append(" operation,\n") - .append(" row_data::text,\n") - .append(" pk_data::text\n") + .append(" row_data::jsonb,\n") // Ensure it's passed as valid JSONB + .append(" pk_data::jsonb\n") // Ensure primary key is passed as JSONB .append(" )\n") .append(" );\n") .append(" RETURN NEW;\n") .append("END;\n") .append("$$ LANGUAGE plpgsql;\n"); + + // Log the SQL statement for debugging purposes System.out.println(functionSql.toString()); + // Execute the SQL to create the function jdbcTemplate.execute(functionSql.toString()); } diff --git a/src/main/java/com/example/yuga/start/service/PostgresNotificationService.java b/src/main/java/com/example/joel/start/service/PostgresNotificationService.java similarity index 81% rename from src/main/java/com/example/yuga/start/service/PostgresNotificationService.java rename to src/main/java/com/example/joel/start/service/PostgresNotificationService.java index 535108738..780196174 100644 --- a/src/main/java/com/example/yuga/start/service/PostgresNotificationService.java +++ b/src/main/java/com/example/joel/start/service/PostgresNotificationService.java @@ -1,8 +1,7 @@ -package com.example.yuga.start.service; +package com.example.joel.start.service; -import com.example.yuga.start.gsheet.SheetsService; -import com.example.yuga.start.repos.SchemaRegistryRepository; -import org.postgresql.PGConnection; +import com.example.joel.start.gsheet.SheetsService; +import com.example.joel.start.repos.SchemaRegistryRepository; import org.postgresql.PGNotification; import org.postgresql.jdbc.PgConnection; import org.springframework.jdbc.core.JdbcTemplate; @@ -54,16 +53,17 @@ private void listenForChanges() { // System.out.println("Received notification: " + notification.getParameter().formatted()); String ans = notification.getParameter(); System.out.println("ans = "+ans); - SqlNotificationPayload sqlNotificationPayload = SqlNotificationPayload.fromNotification(ans); + SqlNotificationPayload sqlNotificationPayload = SqlNotificationPayload.fromNotification(notification.getParameter()); // Further processing logic (e.g., trigger service or handle data) - System.out.println(sqlNotificationPayload.toString()); + System.out.println("sql="+ sqlNotificationPayload); Map mapping =schemaRegistryRepository.getGSheetColumnMapping(sqlNotificationPayload); - if(sqlNotificationPayload.getType().equals("DELETE")){ - sheetsService.deleteRowIfExists("1dexgjeGyQt4ZBq8Z0fCaQRAIX7ARZKUkHMmNjRkgUTY",mapping); + String sheetId = schemaRegistryRepository.getLink(sqlNotificationPayload.getTableName()); + if(sqlNotificationPayload.getOperation().equals("DELETE")){ + sheetsService.deleteRowIfExists(sheetId,mapping); } else { - sheetsService.insertData("1dexgjeGyQt4ZBq8Z0fCaQRAIX7ARZKUkHMmNjRkgUTY", mapping); + sheetsService.insertData(sheetId, mapping); } } @@ -73,7 +73,7 @@ private void listenForChanges() { Thread.sleep(500); } } catch (Exception e) { - e.printStackTrace(); +// e.printStackTrace(); } } diff --git a/src/main/java/com/example/yuga/start/service/SchemaRegistryService.java b/src/main/java/com/example/joel/start/service/SchemaRegistryService.java similarity index 94% rename from src/main/java/com/example/yuga/start/service/SchemaRegistryService.java rename to src/main/java/com/example/joel/start/service/SchemaRegistryService.java index 092c195bd..52afc5d9b 100644 --- a/src/main/java/com/example/yuga/start/service/SchemaRegistryService.java +++ b/src/main/java/com/example/joel/start/service/SchemaRegistryService.java @@ -1,9 +1,9 @@ -package com.example.yuga.start.service; +package com.example.joel.start.service; +import com.example.joel.start.repos.SchemaRegistryRepository; import org.springframework.stereotype.Service; import java.util.List; import java.util.Map; -import com.example.yuga.start.repos.SchemaRegistryRepository; @Service public class SchemaRegistryService { diff --git a/src/main/java/com/example/joel/start/service/SqlNotificationPayload.java b/src/main/java/com/example/joel/start/service/SqlNotificationPayload.java new file mode 100644 index 000000000..b09a32428 --- /dev/null +++ b/src/main/java/com/example/joel/start/service/SqlNotificationPayload.java @@ -0,0 +1,78 @@ +package com.example.joel.start.service; + +import org.json.JSONObject; +import java.util.HashMap; + +public class SqlNotificationPayload { + private String tableName; + private String operation; + private HashMap data; + private String primaryKey; + + public static SqlNotificationPayload fromNotification(String notification) { + SqlNotificationPayload payload = new SqlNotificationPayload(); + + String[] parts = notification.split(" - "); + for (String part : parts) { + String[] keyValue = part.split(": ", 2); + if (keyValue.length == 2) { + String key = keyValue[0].trim(); + String value = keyValue[1].trim(); + + switch (key) { + case "Table": + payload.setTableName(value); + break; + case "Operation": + payload.setOperation(value); + break; + case "Data": + payload.setData(parseJson(value)); + break; + case "PrimaryKey": + payload.setPrimaryKey(parseJson(value).get("id")); + break; + } + } + } + + return payload; + } + + private static HashMap parseJson(String jsonString) { + HashMap map = new HashMap<>(); + try { + JSONObject json = new JSONObject(jsonString); + for (String key : json.keySet()) { + map.put(key, json.getString(key)); + } + } catch (Exception e) { + // Log the error or handle it as needed +// e.printStackTrace(); + } + return map; + } + + @Override + public String toString() { + return "SqlNotificationPayload{" + + "tableName='" + tableName + '\'' + + ", operation='" + operation + '\'' + + ", data=" + data + + ", primaryKey='" + primaryKey + '\'' + + '}'; + } + + // Getters and setters + public String getTableName() { return tableName; } + public void setTableName(String tableName) { this.tableName = tableName; } + + public String getOperation() { return operation; } + public void setOperation(String operation) { this.operation = operation; } + + public HashMap getData() { return data; } + public void setData(HashMap data) { this.data = data; } + + public String getPrimaryKey() { return primaryKey; } + public void setPrimaryKey(String primaryKey) { this.primaryKey = primaryKey; } +} \ No newline at end of file diff --git a/src/main/java/com/example/yuga/start/service/SqlNotificationPayload.java b/src/main/java/com/example/yuga/start/service/SqlNotificationPayload.java deleted file mode 100644 index 64d9c4773..000000000 --- a/src/main/java/com/example/yuga/start/service/SqlNotificationPayload.java +++ /dev/null @@ -1,102 +0,0 @@ -package com.example.yuga.start.service; - -import org.json.JSONObject; -import java.util.HashMap; - -public class SqlNotificationPayload { - private String type; - private HashMap data; - private String pkey; - private String tableName; - // Getters and setters - public String getType() { - return type; - } - - public void setType(String type) { - this.type = type; - } - - public HashMap getData() { - return data; - } - - public void setData(HashMap data) { - this.data = data; - } - - public String getPkey() { - return pkey; - } - - public void setPkey(String pkey) { - this.pkey = pkey; - } - - public String getTableName() { - return tableName; - } - - public void setTableName(String tableName) { - this.tableName = tableName; - } - - // Method to parse notification and fill SqlNotificationPayload - public static SqlNotificationPayload fromNotification(String notification) { - SqlNotificationPayload payload = new SqlNotificationPayload(); - - String[] parts = notification.split(" - ", 4); // Split into 4 parts maximum - if (parts.length >= 3) { - // Extract table name - String[] tableParts = parts[0].split(":", 2); - if (tableParts.length == 2) { - payload.setTableName(tableParts[1].trim()); - } - - // Extract operation type - String[] operationParts = parts[1].split(":", 2); - if (operationParts.length == 2) { - payload.setType(operationParts[1].trim()); - } - - // Extract data and primary key - String dataAndPkPart = parts[parts.length - 1]; // Last part contains data and primary key - String[] dataParts = dataAndPkPart.split("PrimaryKey:", 2); - if (dataParts.length == 2) { - String dataJson = dataParts[0].replace("Data:", "").trim(); - String pkJson = dataParts[1].trim(); - - HashMap dataMap = new HashMap<>(); - - // Parse Data JSON - JSONObject dataObject = new JSONObject(dataJson); - dataObject.keys().forEachRemaining(key -> dataMap.put(key, dataObject.getString(key))); - - // Parse PrimaryKey JSON - JSONObject pkObject = new JSONObject(pkJson); - if (pkObject.length() > 0) { - String pkKey = pkObject.keys().next(); - String pkValue = pkObject.getString(pkKey); - payload.setPkey(pkValue); - // Also add the primary key to the data map if it's not already there - if (!dataMap.containsKey(pkKey)) { - dataMap.put(pkKey, pkValue); - } - } - - payload.setData(dataMap); - } - } - - return payload; - } - - @Override - public String toString() { - return "SqlNotificationPayload{" + - "type='" + type + '\'' + - ", data=" + data + - ", pkey='" + pkey + '\'' + - '}'; - } -} diff --git a/src/test/java/com/example/yuga/start/StartApplicationTests.java b/src/test/java/com/example/joel/start/StartApplicationTests.java similarity index 84% rename from src/test/java/com/example/yuga/start/StartApplicationTests.java rename to src/test/java/com/example/joel/start/StartApplicationTests.java index 0ba836acc..ccd103cb3 100644 --- a/src/test/java/com/example/yuga/start/StartApplicationTests.java +++ b/src/test/java/com/example/joel/start/StartApplicationTests.java @@ -1,4 +1,4 @@ -package com.example.yuga.start; +package com.example.joel.start; import org.junit.jupiter.api.Test; import org.springframework.boot.test.context.SpringBootTest; From 34b32c3f2e8566735dd6bba5e14ef91785a7f6b9 Mon Sep 17 00:00:00 2001 From: Joel Renjith Date: Tue, 17 Sep 2024 02:48:33 +0530 Subject: [PATCH 19/23] Initial commit --- .gitignore | 1 + src/main/resources/application.properties | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.gitignore b/.gitignore index 549e00a2a..785855e2d 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,4 @@ build/ ### VS Code ### .vscode/ +credentials.json \ No newline at end of file diff --git a/src/main/resources/application.properties b/src/main/resources/application.properties index a2e51c39b..75d750cbb 100644 --- a/src/main/resources/application.properties +++ b/src/main/resources/application.properties @@ -16,9 +16,9 @@ spring.jpa.generate-ddl=true # ------------------- # Data-source config. spring.sql.init.platform=postgres -spring.datasource.url=jdbc:postgresql://localhost:5432/super -spring.datasource.username=postgres -spring.datasource.password=JoeL@123! +spring.datasource.url=jdbc:postgresql://localhost:5432/ +spring.datasource.username= +spring.datasource.password= # HikariCP config (pool size, default isolation level). spring.datasource.type=com.zaxxer.hikari.HikariDataSource From 73b0a1d8d9e83428971a4c9029f426fc71e762b9 Mon Sep 17 00:00:00 2001 From: Joel Renjith Date: Tue, 17 Sep 2024 02:51:06 +0530 Subject: [PATCH 20/23] Remove sensitive file --- credentials.json | 13 ------------- 1 file changed, 13 deletions(-) delete mode 100644 credentials.json diff --git a/credentials.json b/credentials.json deleted file mode 100644 index c5da59701..000000000 --- a/credentials.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "type": "service_account", - "project_id": "gsheetssqlmap", - "private_key_id": "74fa4a5248c5447a3084ade43d3cf4469a575b2d", - "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDiK8F7/QvFjMFD\nIV5h0m6o2TjShSTT0HiDsNzN9qiIzijBAnnwg/525GxMvpK7yi0SqHYC3jvjR2FH\nRv6RighkayAeoZ2vwe7Gj0I0Rix5PHnXsw+/MmkshiKATyGLaDdf/Ihyb0K+Eaod\n2uecaCoWGFXiwYlD1MZzxRK/Nd4mMA1afQQcxr8l5YwtbuhlWuvKnUN5iJ1ievuF\nNJa2fKw9aYQD+7tL88V3rvLgtky3wVBNHbgWvO3piEg4E9oQzCmqF7fRuX+4Lggj\nxJAIoDHi0r1vsvfqFAL2dXPBAARvNwIjVlck5U134QlVoznx4LmvN9tspBVmxFgN\neY1H/jtPAgMBAAECggEARVjSOhCkD93xEgLEvCy4BmiYPiBZ4uJMWWCcy0Aie9Pu\ndgOq3N6z4sN2G2/hcNf0SwDBMm7uoOjtgFvZyUkv+o5IKmw2e8H0Uwhw7bNIhqHO\n4B7EqMStr9uu73WT3PlG5VT6h0/rAHlNFltc4b2kjeLHTsA1tMkkUjHuZhd/tmg4\nz+0F9nI82eODL0hYNII8OwBEoXXSj9687fIDwF96W45hoFFXftons8agVu4qHjdc\nZZsU28zZTXw8TJLrgN0Rql9VOUnDRNBBrVLzzcU0dccOkg9dbsCZknQCdv+IF2gg\nC/rcBfwWKtSyonhPttR3UM7szYMkUWa5hG3aZMvMLQKBgQD16TTeeOLui9r1orrb\nQ9UwA+BnLRz7MWqiA2PUw4cM2BlM0kJWRMzp0SiKAilzSOffbC14HZgO1SmqcZ1I\n/2q9wghGvoyXqGjDRWfJJE0ndhNCR3KxVbyRpt0HT72JXaaH5Ru7fOga+/fG/Mi4\n/eXgFoeCoLehm+C4VldroQQ+awKBgQDrczhqL+XWMWEVgmJErak3mQvj5rm9s2hI\nRjABFsZFsB/y0tNnw7fUdbIPpTiUz8LnHGBVzEp4vIjWuX9jVBDuWZ96CSsg6Xly\nvLvqiPeVySZJcEulZNqBbrYaHiHsS15T7uL6LL/IdUNA6wfeKXp4GGB5Jm2MAqaW\nqWoJg8lnrQKBgGUff1Q1FezOqj30qy5bmKFkFrWXAaHyUk1JY5Ysya5odw72L71W\nEYXdMvR9qxnzUxa3X4NTeCIxWi7zBaDtF7HsxoaA+UOsR1h1hBZczTV7ArRwuSNf\nhGnI2n+VNPX2AvNDLcJNvQzLrLqxXMFl90/NM/BtstXMSf7D0rM5xYr9AoGBAJ09\n/dcs1doOcD/lK6uiLviqhX5WgIVoY6Sg+z4+73tq/Gs20WjjFMoXMCuys+7uMcQW\n919VaAbqIpkEmUbExenogNEHfn4kbq9pzDX98xtdkuxIyrxSdnsqbQNUTGN6kiyL\nxZo+eHZvmdKpJEACg3/hB8zhFRgETS0hR2fDU6ahAoGBAMRVsxNkBSDJwb2vNSED\nTCFSLFdPQl4YzMhwonD/8fmr2yfroLGFD+VDCC5vu2a0XXyxfbRtMKAGC1eQXVIU\nHTV9dA7aUgyWKtb54QWZ1M419lF/7oQIMRE3B9eS+UhtRIaxVW9UFZvsuaCjUf5w\n2XpPJyzQ1+VbpRzneo03KaMV\n-----END PRIVATE KEY-----\n", - "client_email": "gmapsheet@gsheetssqlmap.iam.gserviceaccount.com", - "client_id": "108264969693417699931", - "auth_uri": "https://accounts.google.com/o/oauth2/auth", - "token_uri": "https://oauth2.googleapis.com/token", - "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", - "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gmapsheet%40gsheetssqlmap.iam.gserviceaccount.com", - "universe_domain": "googleapis.com" -} From 715bd1e9a4f09a7f663604cc7192afc6b5b2b6df Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 02:54:38 +0530 Subject: [PATCH 21/23] Create App Script --- App Script | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 App Script diff --git a/App Script b/App Script new file mode 100644 index 000000000..79206fa12 --- /dev/null +++ b/App Script @@ -0,0 +1,115 @@ + +function extractSheetId(url) { + // Regular expression to match the sheet ID in the Google Sheets URL + var regex = /\/d\/(.*?)\/edit/; + var match = url.match(regex); + + if (match && match[1]) { + return match[1]; // Return the sheet ID + } else { + throw new Error("Sheet ID not found in URL."); + } +} + +function sendNotification(e) { + var url =SpreadsheetApp.getActiveSpreadsheet().getUrl();; +try { + var sheetId = extractSheetId(url); + console.log("Sheet ID:", sheetId); +} catch (error) { + console.error(error.message); +} + console.log("sendNotification function called"); + if (!e) { + console.error("Event object is undefined"); + return null; + } + if (!e.source) { + console.error("Event object is missing 'source' property"); + return null; + } + if (!e.range) { + console.error("Event object is missing 'range' property"); + return null; + } + + try { + var sheet = e.source.getActiveSheet(); + if (!sheet) { + console.error("Unable to get active sheet"); + return null; + } + + var range = e.range; + console.log("Range:", range.getA1Notation()); + + var newValue, oldValue, columnName, id, timestamp; + try { + newValue = range.getValue(); + console.log("New value:", newValue); + } catch (error) { + console.error("Error getting new value:", error.toString()); + } + + try { + oldValue = e.oldValue || null; + console.log("Old value:", oldValue); + } catch (error) { + console.error("Error getting old value:", error.toString()); + } + + try { + var column = range.getColumn(); + columnName = sheet.getRange(1, column).getValue(); + console.log("Top cell value in column:", columnName); + } catch (error) { + console.error("Error getting top cell value:", error.toString()); + } + + try { + id = sheet.getRange(range.getRow(), 1).getValue(); + console.log("Value at Column A:", id); + } catch (error) { + console.error("Error getting column A value:", error.toString()); + } + + timestamp = new Date().toISOString(); + console.log("Timestamp:", timestamp); + + // var sheetId = sheet.getSheetId(); + var payload = { + sheetid: sheetId, + columnName: columnName, + oldValue: oldValue, + newValue: newValue, + id: id, + timestamp: timestamp + }; + + console.log("Payload prepared:", JSON.stringify(payload)); + + var url = '/notifications'; + + var options = { + 'method': 'POST', + 'contentType': 'application/json', + 'payload': JSON.stringify(payload) + }; + + try { + var response = UrlFetchApp.fetch(url, options); + console.log("Response status:", response.getResponseCode()); + console.log("Response content:", response.getContentText()); + } catch (error) { + console.error("Error sending request:", error.toString()); + } + + console.log("POST request sent to:", url); + + return sheetId; // Return the sheet ID + + } catch (error) { + console.error("Error in sendNotification:", error.toString()); + return null; + } +} From b887c947eb757fe670bb0065ceac6852d9f27cfa Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 16:47:01 +0530 Subject: [PATCH 22/23] postgres --- README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/README.md b/README.md index 874ff9566..7ab291de1 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,23 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- ### Application-Setup: * Visit google marketplace > search for sheets > click on the google sheets api and click enable > click on credentails and add a service account > click on the service account and select keys > create a new key, download it and put the JSON file into the directory right outside the src directory in the project. * in the application.properties file enter your postgres database name in the url and your postgres username and password. +* go to your postgres database and enter the following commands + - ``` + CREATE TABLE public.schema_link ( + id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + link VARCHAR(255), + table_name VARCHAR(255) + ); + ``` + - ``` + CREATE TABLE public.schema_registry ( + id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + table_name VARCHAR(255), + column_name VARCHAR(255), + column_type VARCHAR(255), + gsheetcol VARCHAR(10) + ); + ``` ### User-Setup: * The user has to make a Google Sheet and change access from "Restricted" to "Anyone with the link" and set permissions to "Editor" From 37f0810aa2d166fbe53faa624c62563dca5bee81 Mon Sep 17 00:00:00 2001 From: Joel Renjith <98686646+joelrenjith@users.noreply.github.com> Date: Tue, 17 Sep 2024 17:05:59 +0530 Subject: [PATCH 23/23] edge case --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7ab291de1..aa850ac15 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ We have a checklist at the bottom of this README file, which you should update a - [x] My code's working just fine! 🥳 - [x] I have recorded a video showing it working and embedded it in the README ▶️ - [x] I have tested all the normal working cases 😎 -- [ ] I have even solved some edge cases (brownie points) 💪 +- [x] I have even solved some edge cases (brownie points) 💪 - [x] I added my very planned-out approach to the problem at the end of this README 📜 ## Got Questions❓ @@ -60,7 +60,7 @@ All the best ✨. ## Developer's Section ### Architectue of the application: -My application is built using Spring Boot and PostgreSQL, providing robust real-time synchronization between multiple pairs of database tables and Google Sheets. The system architecture includes the following key components: +My application is built using Spring Boot and PostgreSQL, providing real-time synchronization **between multiple pairs of database tables and Google Sheets**. The system architecture includes the following key components: * **Real-Time Synchronization:** @@ -82,6 +82,10 @@ My application is built using Spring Boot and PostgreSQL, providing robust real- * _Remote Access_: The Spring Boot application is exposed to the internet using ngrok, allowing external access and interaction with the application. +**Edge Cases:** +* As mentioned above, my application can handle synchronisation _between multiple pairs of database tables and Google Sheets_. +* The linking between tables in the database and Gooogle Sheets is persistent, i.e, the previous link between a Sheet and a Table remains connected even if the application has resatarted, while also being ready to create new links between another table and sheet on user demand. + ### Requirements for the Setup: * **Java Development Kit (JDK):**