Developer manual

What's that?

Tools for importing data from a sensor directly on the istSOS server. istSOS is a python based server for maintaining sensor observation services.

On istSOS you can manage and dispatch your sensor data with graphical user interface or through python scripts. istSOS works as a storage of your data as well as a medium for visualizing or analyzes.

What does mean SOS? It's sensor observation service, an OGC format standard for providing sensor observation data and even for some operations with them.

SOS standard has very special vocabulary. So let's explain what all those strange words means:

Service = Type of logger device (Temperatur logger, Moisture logger... )
Offering = Grouping unit for procedures (e.g. Project)
Procedure = Individual sensor or other individual device writing measurements

Getting started

What you need and what shall you have installed before you start your work with istSOS?

istSOS

A python based server for maintaining sensor observation services. On istSOS you can manage and dispatch your sensor data with graphical user interface or through python scripts.

See the project documentation here[1].

PostgreSQL

Object-relational database management system working with a database server.

See the project documentation here[2].

PostGIS

PostgreSQL support for geographic data.

See the project documentation here[3].

pgAdmin

Graphical user interface for PostgreSQL.

See the project documentation here[4].

Apache

Server providing HTTP services.

See the project documentation here[5].

libapache2-mod-wsgi

Python WSGI (interface between web server software and web applications) module for Apache.

See the project documentation here[6].

psycopg2

Python interface for PostgreSQL.

See the project documentation here[7].

How to use these tools?

Manuals are written for Linux OS.

Clone this repository

To use these tools, you have to clone this repository to your machine, it means make a copy of this repository on your local computer.

It can be done by one simple command ran from your desired folder:

git clone https://github.com/NINAnor/istSOS-support.git

Go to the new directory istSOS-support. Mostly:

cd istSOS-support

Then you can run freely all the scripts from this place.

Make your server prepared for the data

Before you start your data loading, you have to make your server prepared for them. It means that you need some server, some service(s) on that server, some offering(s) in those services and procedure(s) in those offerings.

You have to do it manually in the web interface (or most of that). The best way is to follow the official istSOS manual[8]. Or it is at least the best way until you get to the procedures registration, because this step can be done in much faster way using automatized python scripts.

For procedures registration with procedures2istsos, you have to have prepare an geometry index file for procedures and procedures themselves. One geometry index file template prepared for NINA templogger service can be found here in the repository.

Then, you just have to run the script with your own parameters (instead of asterisks):

python procedures2istsos.py -path=* -url=* --help

TIP: Need help? Don't know what these parameters mean? No problem, just run this command: python procedures2istsos.py --help

Quick way using bash script

Run shell script with your own parameters (instead of asterisks):

bash import2istsos.sh file_path=* observation_columns=* timestamp_column=* timestamp_format=* url=* service=* offering=* procedure=*

If you got message saying > Insert observation success: True, your data were imported and everything was OK.

NOTE: A .dat file(s) was (were) created in your file directory. If you wish to delete it, you have to do it yourself.

TIP: Need help? Don't know what these parameters mean? No problem, just run this command: bash import2istsos.sh --help

Slower, but more thorough and much cooler way

Clone this github repository to your computer:

git clone https://github.com/NINAnor/istSOS-support.git

Go to the new directory istSOS-support. Mostly:

cd istSOS-support

Run python script with your own parameters (instead of asterisks):

python convert2dat.py -path=* -observation_columns=* -timestamp_column=* -timestamp_format=*

If there is no warning message, your data were converted into istSOS acceptable format in .dat file in the same directory as your original file was.

Now go to the istSOS directory. Mostly:

cd /usr/share/istsos/

Run python script with your own parameters (instead of asterisks):

python scripts/csv2istsos.py -u * -s * -o * -p * -w * -m *

If you got message saying > Insert observation success: True, your data were imported and everything was OK.

NOTE: A .dat file(s) was (were) created in your file directory. If you wish to delete it, you have to do it yourself.

TIP: Need help? Don't know what these parameters mean? No problem, just run one of these commands: python convert2dat.py --help or python scripts/csv2istsos.py --help

Yes, but what to do with those data?

You can just store them. Or you can do some analyzes above them. How? And which analyzes?

Using istSOS

You can visualize your data on the istSOS serverin the form of charts or maps using the data viewer. You can also visualize them with more advanced approach - adjust maps according to your needs following this manual or you can get modified observations like here.

Everything offer also aggregations and other data management.

Using GRASS GIS

Yes, handling everything online is really cool. But sometimes you need more sophisticated tools, more specific environment, more and more things. In these cases, there is also a possibility to use GRASS GIS.

Firstly, you must extract data from your istSOS server and import them into GRASS GIS. So start GRASS GIS. Then you can import data directly from istSOS right now in three ways using AddOns from this repository.

Import an offering as a vector map with layer for each observed property:

v.in.sos url=* output=* offering=*

Import an offering as a set of raster maps for each timestamps:

r.in.sos url=* output=* offering=*

Import an offering as time series of vector layers for each observed property:

t.vect.in.sos url=* output=* offering=*

All these AddOns have of course their own helps by typing command:

*.in.sos --help

If no error occurs, your data were successfully imported into GRASS GIS. You can do whatever GRASS GIS allows you to do with them (even convert them into other format and work with them for example in QGIS).

Example

I want to upload ids and temperatures to the server localhost/istsos/, on service myservice and offering temporary.

My sample data

Repository cloned in /home/user/workspace/

Data are saved here: /home/user/Documents/myprocedure_20172807175100.csv

timestamp	id	garbage	urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature
2017-28-07T15:51:00.000000+01:00	1	fox	24.5
2017-28-07T16:51:00.000000+01:00	2	moose	25
2017-28-07T17:51:00.000000+01:00	3	manatee	3.14

Import

Using shell script

cd /home/user/workspace/istSOS-support/
bash import2istsos.sh file_path=/home/user/Documents/myprocedure_20172807175100.csv observation_columns=id,urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature timestamp_column=timestamp timestamp_format=YYYY-MM-DDTHH:MM:SS.SSSSSS+HH:MM url=http://localhost/istsos service=myservice offering=temporary procedure=myprocedure

Output:

creating .dat from your file
done

uploading your .dat file on the server
Offering: temporary
Procedure: myprocedure
myprocedure > Sensor Description successfully loaded
myprocedure > GetObservation requested successfully executed
Searching: /home/user/Documents/myprocedure_[0-9]*.dat
Before insert ST: myprocedure
 > Begin: 2010-01-01T00:00:00+01:00
   + End: 2017-31-12T23:59:59+01:00
Insert ST: myprocedure
 > Begin: 2017-28-07T15:51:00+01:00
   + End: 2017-28-07T17:51:00+01:00
 > Values: 3
 > Insert observation success: True

TIP: You can also use -d flag, parameter file_extension=csv and file_path=/home/user/Documents/myprocedure to parse all .csv files beginning at myprocedure in the Documents folder.

Using python scripts

python /home/user/workspace/istSOS-support/convert2dat.py -path=/home/user/Documents/myprocedure_20172807175100.csv -observation_columns=id,urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature -timestamp_column=timestamp -timestamp_format=YYYY-MM-DDTHH:MM:SS.SSSSSS+HH:MM
cd /usr/share/istsos/
python scripts/csv2istsos.py -u http://localhost/istsos -s myservice -o temporary -p myprocedure -w /home/user/Documents/ -m 1000