DataScoop

Imports data from one or more csv files and inserts them into a target postgres table. Postgres is currently supported, but other database technologies will be followed in the future.

Import Preset File

CSV files can be specified as a datasource grouped by there structure. A mapping must be stored in a json configuration file for each structure. This mapping applies to all files to be imported per datasource. The configuration file is called import preset file.

An example for an import preset file would be:

{
  "datasources": {
    "datasource-1": {
      "csv-files": [
        "/tmp/example1.csv",
        "/tmp/example2.csv"
      ],
      "csv-separator": ",",
      "csv-header": true,
      "csv-encoding": "UTF-8",
      "csv-mapping": {
        "Name": "text",
        "Age": "integer",
        "City": "text",
        "Email": "text",
        "Salary": "float",
        "Active": "boolean"
      }
    }
  },
  "database": {
    "url": "jdbc:postgresql://localhost:5432/datascoop",
    "username": "datascoop",
    "password": "datascoop",
    "table-name": "target_table_name"
  }
}

A corresponding CSV file would be (data was generated):

Name,Age,City,Email,Salary,Active
fnvpUxID,59,wbvdVd,[email protected],3216.54,True
bClkrBnT,52,lDpIkT,[email protected],2837.87,True
XSvpCdbm,26,ieApnz,[email protected],3109.29,True
mmZlkWvB,54,TvjQCK,[email protected],1802.94,True
CJNTOZtB,31,tGanpg,[email protected],3222.58,False
phWMCYSZ,18,StcWnh,[email protected],2656.52,False
iCeMWXct,56,saZhxa,[email protected],3024.28,True
vLKLEjZQ,48,RtyCDE,[email protected],2945.15,True
NvxqUUua,51,WAwOrX,[email protected],3741.78,True

Data Types

The following data types can be used in the import preset file:

Data Type	Description
text	This is the most universal data type. It can either be a simple text or a column that cannot be parsed or represented by the other data types.
integer	A number without a decimal place
float	A number with a decimal place
boolean	A bool value can either be True, False or empty

Build and Run

You can either create and run the project with your own gradle version or with the gradlew wrapper which comes with this project. In both cases, please open a terminal in the root directory of the project. Then execute one of the following options:

• gradlew: Use cli and go into the root project folder, then execute: ./gradlew jar or ./gradlew.bat jar on win
• own gradle installation gradle jar

After the build process is complete, you can find the JAR executable file in the build/libs directory of the project. You can then use this JAR file to run the app.

java -jar build/libs/datascoop-0.0.1-SNAPSHOT.jar [path/to/your/import_preset_file]