Skip to content

Lonoa User Guide

Jump to bottom
matthew-schultz edited this page Dec 22, 2019 · 13 revisions

Sections

Initial_setup

In order to set up a new lonoa project see the readme https://github.com/erdl/lonoa/blob/master/README.md

Permissions

File permissions

  1. lonoa
    • Running a lonoa project requires the lonoa user to have read, write, and execute file permissions on that project's folder.
    • Run the command su lonoa on the server before running setup scripts like init_database.py, because it substitutes in the lonoa user, ensuring that those setup scripts create the project with the proper file permissions.
  2. general users
    • in order to upload hobo csv's, general users will need a way to move csv files into the folder <lonoa project name>/hobo/script/to-insert/ which can be done 2 main ways
      • method 1 upload the file to a directory they can write to and copy the folder after running su lonoa
      • method 2 obtain read/write permissions on the folder <lonoa project name>/hobo/script/to-insert/

Database permissions

  1. lonoa user - needs create database permissions
    • psql -c 'CREATE USER lonoa WITH CREATEDB;' <database name>
  2. general users - can add them to lonoa postgres database group, so they can access the database
    • psql -c 'ALTER GROUP lonoa ADD USER <username>;' <database name>

Adding_new_sensors

Adding new sensors with existing script_folders

Add new sensors to the sensor_info table, in order to start acquiring their readings.

egauge

in order to run api_egauge.py, egauge rows in sensor_info need these columns to be filled

  • purpose_id
  • query_string - first part of egauge api url; (e.g. egaugeX in http://egaugeX.egaug.es/)
  • script_folder - egauge
  • unit - e.g. kW, F, RH, etc.
  • data_sensor_info_mapping - column name in egauge reading csv obtained from api; e.g. Usage [kW]
  • is_active - True
  • last_updated_datetime - datetime 1 minute before the first reading you want to obtain

webctrl

in order to run api_webctrl.py, webctrl rows in sensor_info need these columns to be filled to run

  • purpose_id
  • query_string
  • script_folder - webctrl
  • unit - e.g. kW, F, RH, etc.
  • is_active - True
  • last_updated_datetime - datetime 1 minute before the first reading you want to obtain

hobo

in order to run extract_hobo.py, hobo rows in sensor_info need these columns to be filled to run

  • purpose_id
  • query_string - value after plot title in hobo csv (e.g. 9790163 in "Plot Title: 9790163")
  • script_folder - hobo
  • unit - e.g. kW, F, RH, etc.
  • is_active - True You will need a row in sensor_info for each unit column in the hobo csv (which usually has 3 columns), in order for any of its readings to be inserted.

Adding a new sensor after adding a new script_folder + data acquisition script

Prerequirements

You should already have created a new script folder with a working data acquisition script

lonoa/<script folder>/script/api_<script_folder>.py

  • ex. lonoa/raspberry_pi/script/api_raspberry_pi.py

1. Update sensor_info.script_folder column (with Postgres enum datatype) in database

Add new value to enum

ALTER TYPE <enum class name> ADD VALUE '<value>';

  • ex. ALTER TYPE scriptfolderenum ADD VALUE 'raspberry_pi';

2. Add new script_folder value to orm python file's ScriptFolderEnum class

3. Add new sensors to sensor_info with appropriate values set

Running_scripts_manually

For the examples below, the project's folder name is lonoa_frogs.

Before running lonoa manually using the commands below you should:

  1. ssh into the server
  2. run the command su lonoa (to substitute in the lonoa user, which should have the proper permissions to run the scripts)

egauge

cd lonoa_frogs/egauge/script/
python3 api_egauge.py

webctrl

cd lonoa_frogs/webctrl/script/
python3 api_webctrl.py

hobo

cd lonoa_frogs/hobo/script/
python3 extract_hobo.py

Uploading_hobo_csv_files

Upload csv into <project name>/hobo/script/to-insert/

  • ex lonoa/hobo/script/to-insert/

Database_tables

See https://github.com/erdl/lonoa/issues/112#issuecomment-525130397 and https://github.com/erdl/lonoa/blob/master/egauge/script/orm_egauge.py

Troubleshooting

  1. I added a new sensor to the sensor_info table but it's not acquiring data

    • Check that all required fields in sensor_info have been filled
    • Check that is_active column in sensor_info is True
    • Check that the data_sensor_info_mapping column has been filled in correctly for hobo and egauge -needs to exactly match csv column name in: -csv data returned by egauge api -in hobo csv -Try to run the script manually and see if it runs successfully or returns any errors
      • STEPS
        • ssh into server
        • su lonoa
        • cd /home/lonoa//<script folder>/script
        • python3 <script name>
      • If it returns no errors and still does not insert data into readings, check to make sure is_active column in sensor_info is set to True
    • Check that there is a crontab job running for that repo and script_folder
      • STEPS
        • ssh into server
        • su lonoa
        • crontab -e
      • If init_crontab.py does not have a job scheduled, run init_crontab.py
        • STEPS
          • ssh into server
          • su lonoa
          • cd /home/lonoa/
          • python3 init_crontab.py

  2. One of my existing sensors in sensor_info is running but not acquiring data

    • Check error_log table (if hobo also check error_log_details table)
      • STEPS
        • ssh into server
        • psql <database_name>
        • SELECT * from error_log WHERE purpose_id= AND was_success=False ORDER BY log_id desc;
        • SELECT * from error_log_details WHERE log_id=<error_log.log_id>;
    • Check error.log file in script_folder
      • STEPS
        • ssh into server
        • su lonoa
        • cd /home/lonoa//<script_folder>/script
        • vim error.log

  3. My data acquisition scripts are running but they keep returning errors

    • HTTPError or ConnectionError -likely due to internet connection issues - data source (webctrl, egauge) may not have been connected to internet - server may not have been connected to the internet
    • IntegrityError
      • Usually caused by trying to insert duplicate readings
        • webctrl - one job might have run so long that a new job was run tried to insert duplicate readings
        • hobo - the file may have some readings already in the database
      • Make sure that sensor_info's last_updated_datetime column is not less than one of the readings in the database
        • if it is less, manually change last_updated_datetime to be equal to or later than latest datetime value in reading table for that purpose_id
    • TypeError
      • May be returned when a value is missing from the database
        • webctrl
          • Check that webctrl api has a row in api_authentication with username and password
    • Exception
      • hobo - readings from csv file might have already been inserted into the database

Notes

  1. Readings with purpose_id's 98 and 111 are not currently used (see https://github.com/erdl/lonoa/issues/96)
  2. Project repository's folder name should not be renamed after deployment without updating database
    • is initially set by init_database.py if 'project' table has no values in the column project_folder_path
    • Steps to rename folder:
      • 1 deactivate active lonoa sensors
        • update sensor_info set is_active=False where is_active=True;
      • 2 deactivate lonoa crontab jobs
        • wait for init_crontab to run
        • OR run init_crontab.py
          • python3 <path to project repo>/<project repo>/init_crontab.py
      • 3 rename the project repository's folder
      • 4 run init_database.py
        • python3 <path to project repo>/<project repo>/init_database.py <project name>
      • 5 reactivate desired sensors
        • update sensor_info set is_active=True where <some condition>;
      • 6 reactivate lonoa crontab jobs
        • wait for init_crontab to run
        • OR run init_crontab.py
          • python3 <path to project repo>/<project repo>/init_crontab.py
  3. The database columns sensor_info.script_folder and error_log.pipeline_stage both use the postgres datatype enum (https://www.postgresql.org/docs/9.5/datatype-enum.html).
    • The name of the enum in the postgres database corresponds to its class name in the lonoa orm file.
    • You will likely need to modify the script_folder enum (in the project database) and the ScriptFolderEnum class in the lonoa orm file whenever you add a new script folder for a new sensor type.
    • get values in enum
      • SELECT enum_range(NULL::<enum name>);
        • ex: SELECT enum_range(NULL::PipelineStageEnum);
    • add value to enum
      • ALTER TYPE <enum name> ADD VALUE '<value>';
        • ex: ALTER TYPE ScriptFolderEnum ADD VALUE 'raspberry_pi';
    • drop enum (probably also will need to drop tables)
      • DROP TYPE <enum name>;
        • ex: DROP TYPE PipelineStageEnum;
Clone this wiki locally