Developer manual

What's that?

Tools for importing data from a sensor directly on the istSOS server. istSOS is a python based server for maintaining sensor observation services.

On istSOS you can manage and dispatch your sensor data with graphical user interface or through python scripts. istSOS works as a storage of your data as well as a medium for visualizing or analyzes.

What does mean SOS? It's sensor observation service, an OGC format standard for providing sensor observation data and even for some operations with them.

SOS standard has very special vocabulary. So let's explain what all those strange words means:

Service = Type of logger device (Temperatur logger, Moisture logger... )
Offering = Grouping unit for procedures (e.g. Project)
Procedure = Individual sensor or other individual device writing measurements

Getting started

What you need and what shall you have installed before you start your work with istSOS?

istSOS

A python based server for maintaining sensor observation services. On istSOS you can manage and dispatch your sensor data with graphical user interface or through python scripts.

See the project documentation here[1].

PostgreSQL

Object-relational database management system working with a database server.

See the project documentation here[2].

PostGIS

PostgreSQL support for geographic data.

See the project documentation here[3].

pgAdmin

Graphical user interface for PostgreSQL.

See the project documentation here[4].

Apache

Server providing HTTP services.

See the project documentation here[5].

libapache2-mod-wsgi

Python WSGI (interface between web server software and web applications) module for Apache.

See the project documentation here[6].

psycopg2

Python interface for PostgreSQL.

See the project documentation here[7].

How to use these tools?

Manuals are written for Linux OS.

Data format

Your data are expected to come in some format and architecture. If you think your format should be also supported and you can't retype your data into supported format, please feel free to open an issue for that.

Data structure:

service  
│
└───service offering location
│   └───procedure
│       │   INDEX.SWD
│       │   SD201701.SWD
│       │   SD201702.SWD
│       │   ...
│   
└───service offering location
    └───procedure
        │   INDEX.SWD
        │   SD201701.SWD
        │   SD201702.SWD
        │   ...

or just

service offering location
└───procedure
        │   INDEX.SWD
        │   SD201701.SWD
        │   SD201702.SWD
        │   ...

NOTE: If your location is more than one word, please divide them with spaces or minuses.
NOTE: The name SD2017.SWD is not required. It is required only to hold this structure of data archiving.

INDEX file

If using templogger device, your data should contain also INDEX.SWD. The contain of this file should look similar to this:

SIGN ObservedPropertyName (unit of measuring)

For example:

TMP Temperature (*F)
HMD RH (%)

Observations file example

T2	Temperature (*F)	RH (%)
	*F	%
	TMP	HMD
2011-11-01 01:31	38,5	109,9
2011-11-01 03:31	39,4	109,4
2011-11-01 05:31	39,4	109,9
2011-11-01 07:31	40,5	109,9

NOTE: The first mark (T2) is a name of the procedure (must correspond with a folder name where it is located).
NOTE: You can use dot or commas as a decimal point.
NOTE: Tabulators are used as delimiters.

Clone this repository

To use these tools, you have to clone this repository to your machine, it means make a copy of this repository on your local computer.

It can be done by one simple command ran from your desired folder:

git clone https://github.com/NINAnor/istSOS-support.git

Go to the new directory istSOS-support. Mostly:

cd istSOS-support

Then you can run freely all the scripts from this place.

Make your server prepared for the data

Before you start your data loading, you have to make your server prepared for them. It means that you need some server, some service(s) on that server, some offering(s) in those services and procedure(s) in those offerings.

You have to do it manually in the web interface (or most of that). The best way is to follow the official istSOS manual[8]. Or it is at least the best way until you get to the procedures registration, because this step can be done in much faster way using automatized python scripts.

For procedures registration with procedures2istsos, you have to have prepare an geometry index file for procedures and procedures themselves. One geometry index file template prepared for NINA templogger service can be found here in the repository.

Then, you just have to run the script with your own parameters (instead of asterisks):

python procedures2istsos.py -path=* -url=* --help

TIP: Need help? Don't know what these parameters mean? No problem, just run this command: python procedures2istsos.py --help

The quick method: using data2istsos

Run python script with your own parameters (instead of asterisks):

python data2istsos.py -path=* -device_type=* -url=* -u

This command will upload all observations in your path to corresponding procedures on your server.

If you got messages saying > Insert observation success: True, your data were imported and everything was OK.

NOTE: A .dat files were created in your file directories. If you don't wish to preserve them, just use -f flag.
TIP: Need help? Don't know what these parameters mean? No problem, just run this command: python data2istsos.py --help

The slower method or what are intermediate steps?

Firstly, you have to convert your observations into .dat files. Simply run this python script with your own parameters (instead of asterisks):

python convert2dat.py -path=* -timestamp_column=* -timestamp_format=* -d -t

If there is no warning message, your data were converted into istSOS acceptable format in .dat file in the same directory as your original file was.

Now go to the istSOS directory. Mostly:

cd /usr/share/istsos/

Run python script with your own parameters (instead of asterisks):

python scripts/csv2istsos.py -u * -s * -o * -p * -w * -m *

If you got message saying > Insert observation success: True, your data were imported and everything was OK.

NOTE: A .dat file(s) was (were) created in your file directory. If you wish to delete it, you have to do it yourself.
NOTE: This method can parse only one directory, without subdirectories. To save time, it is recommended to use the first way.
TIP: Need help? Don't know what these parameters mean? No problem, just run one of these commands: python convert2dat.py --help or python scripts/csv2istsos.py --help

Yes, but what to do with those data?

You can just store them. Or you can do some analyzes above them. How? And which analyzes?

Using istSOS

You can visualize your data on the istSOS server in the form of charts or maps using the data viewer[9]. You can also visualize them with more advanced approach - adjust maps according to your needs following this manual[10] or you can get modified observations like here[11].

Everything offer also aggregations and other data management.

Using GRASS GIS

Yes, handling everything online is really cool. But sometimes you need more sophisticated tools, more specific environment, more and more things. In these cases, there is also a possibility to use GRASS GIS.

Firstly, you must extract data from your istSOS server and import them into GRASS GIS. So start GRASS GIS. Then you can import data directly from istSOS right now in four ways using AddOns from this repository[12].

Import an offering as a vector map with layer for each observed property:

v.in.sos url=* output=* offering=*

Import an offering as a set of raster maps for each timestamp:

r.in.sos url=* output=* offering=*

Import an offering as spatio temporal vector dataset for each observed property:

t.vect.in.sos url=* output=* offering=*

Import an offering as spatio temporal raster dataset for each observed property:

t.rast.in.sos url=* output=* offering=*

All these AddOns have of course their own helps by typing command:

*.in.sos --help

If no error occurs, your data were successfully imported into GRASS GIS. You can do whatever GRASS GIS allows you to do with them (even convert them into other format and work with them for example in QGIS).

Example

I want to upload ids and temperatures to the server localhost/istsos/, on service myservice and offering temporary.

My sample data

Repository cloned in /home/user/workspace/

Data are saved here: /home/user/Documents/myprocedure_20172807175100.csv

timestamp	id	garbage	urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature
2017-28-07T15:51:00.000000+01:00	1	fox	24.5
2017-28-07T16:51:00.000000+01:00	2	moose	25
2017-28-07T17:51:00.000000+01:00	3	manatee	3.14

Import

Using shell script

cd /home/user/workspace/istSOS-support/
bash import2istsos.sh file_path=/home/user/Documents/myprocedure_20172807175100.csv observation_columns=id,urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature timestamp_column=timestamp timestamp_format=YYYY-MM-DDTHH:MM:SS.SSSSSS+HH:MM url=http://localhost/istsos service=myservice offering=temporary procedure=myprocedure

Output:

creating .dat from your file
done

uploading your .dat file on the server
Offering: temporary
Procedure: myprocedure
myprocedure > Sensor Description successfully loaded
myprocedure > GetObservation requested successfully executed
Searching: /home/user/Documents/myprocedure_[0-9]*.dat
Before insert ST: myprocedure
 > Begin: 2010-01-01T00:00:00+01:00
   + End: 2017-31-12T23:59:59+01:00
Insert ST: myprocedure
 > Begin: 2017-28-07T15:51:00+01:00
   + End: 2017-28-07T17:51:00+01:00
 > Values: 3
 > Insert observation success: True

TIP: You can also use -d flag, parameter file_extension=csv and file_path=/home/user/Documents/myprocedure to parse all .csv files beginning at myprocedure in the Documents folder.

Using python scripts

python /home/user/workspace/istSOS-support/convert2dat.py -path=/home/user/Documents/myprocedure_20172807175100.csv -observation_columns=id,urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature -timestamp_column=timestamp -timestamp_format=YYYY-MM-DDTHH:MM:SS.SSSSSS+HH:MM
cd /usr/share/istsos/
python scripts/csv2istsos.py -u http://localhost/istsos -s myservice -o temporary -p myprocedure -w /home/user/Documents/ -m 1000