Merge branch 'release/0.10.0'

Conflicts:
	CHANGELOG
	bin/redi.py
	setup.py

.gitignore

@@ -20,9 +20,13 @@ all_form_events.xml
 person_form_event_tree.xml
 person_form_event_tree_with_data.xml
 .figleaf
-
+build/
+dist/
 vagrant/.vagrant/
 vagrant/.vimrc
+vagrant/report.html
 vagrant/tmp/
 vagrant/backup-redcap-*.sql
 vagrant/redcap-backup-*.sql
+vagrant/REDI-*.egg
+

CHANGELOG

@@ -1,3 +1,18 @@
+2014-08-07 v0.10.0
+
+ * Revised configuration architecture
+ * Updated code to better conform to our Logging Guidelines
+ * Added PyCap-based import-export tool
+ * Moved temporary files to dynamically-generated, secure temporary folder
+ * Added tests for REDCap connection and alerts when it fails
+ * Added check for data freshness
+ * Added example configuration content
+ * Improved Python packaging
+ * Cleaned up code to PEP-8 standards
+ * Added configuration file documentation
+ * Added dry-run functionality
+ * Added default configuration values where appropriate
+
 2014-08-05 v0.9.1
 
  * HOTFIX to replace smtp.ufl.edu with configurable option in setup.json

Makefile

@@ -1,13 +1,26 @@
 all: compile
 
 build: compile
+
+egg:
+	python setup.py bdist_egg
+
+egg-test: egg
+	bash scripts/egg-test.bash
+
+bdist:
+	python setup.py bdist
+
+sdist:
+	python setup.py sdist
 compile:
 	python -m compileall bin
 	python -m compileall test
 
 test: tests
 tests:
-	python test/TestSuite.py
+	PYTHONPATH=bin python setup.py test
+	[ ! -d config/rules ] || PYTHONPATH=bin python -munittest discover config/rules
 
 coverage:
 	which figleaf || sudo easy_install figleaf
@@ -19,3 +32,7 @@ clean:
 	rm -rf coverage
 	rm -f .figleaf
 	find . -type f -name *.pyc -print | xargs rm -f
+	rm -rf out/*
+	rm -rf dist
+	rm -rf build
+	rm -rf REDI.egg-info

README.md

@@ -32,6 +32,95 @@ Usage
 
     $python <project_root_path>/bin/redi.py
 
+    Optional command line arguments:
+
+    1. -c : To specify the path to config folder.
+
+            E.g.: $python <project_root_path>/bin/redi.py -c /Users/admin/redi-config
+
+            Default path for config folder is <project_root_path>/config
+
+    2. -k : To keep the output files generated during data processing.
+
+            E.g.: $python <project_root_path>/bin/redi.py -k yes
+
+            When this parameter is provided, the output files are stored in <project_root_path>/out/out_<timestamp>.
+            The timestamp has the format: YYYY_MM_DD-HH_MM_SS
+            If the parameter is not provided, the output files are stored in a temporary folder during the execution of redi.py and then deleted along with the temporary folder once redi.py finishes execution.
+
+    3. -d : To execute redi.py in dry run state.
+
+            E.g.: $python <project_root_path>/bin/redi.py -d
+
+            When this parameter is provided, all data transformations are performed and execution stops after writing out the final data set to <project_root_path>/out/out_<timestamp>.
+            The purpose of this switch is to assist developers in performing a dry run of redi.py without involving the REDCap server and sending of email.
+            If -d is used then we do not need to use -k in addition to it or provide any path for storing the output files.
+            By default, this parameter is disabled.
+
+    4. -e : To run the script for fetching EMR data.
+
+            E.g.: $python <project_root_path>/bin/redi.py -e yes
+
+            When this parameter us provided, a connection will be established with the sftp server mentioned in the settings.ini file in the config folder and EMR data required for the execution of redi.py will be downloaded.
+            Following parameters need to be set in config/settings.ini before using this option:
+            - emr_sftp_server_hostname = URI of sftp server
+            - emr_sftp_server_username
+            - emr_sftp_server_password
+            - emr_sftp_project_name = folder on the sftp server containing the EMR data
+            - emr_data_file = file containing the EMR data
+            - emr_log_file = file to be used for logging
+            By default, this parameter is disabled. It is also disabled if redi is executing in dry run state
+
+About required and optional configuration parameters
+----------------------------------------------------
+
+Following parameters are required to have a value in settings.ini:
+ - raw_xml_file
+ - translation_table_file
+ - form_events_file
+ - research_id_to_redcap_id
+ - component_to_loinc_code_xml
+
+The program will terminate if they are missing or do not have a value in setings.ini. A message about this will be displayed to the user before the program terminates. The same is also written to the log file.
+The above parameters are required irrespective of the state in which redi is executing (normal state or dry run state).
+
+Following parameters are required to have a value in settings.ini when redi is executing in the normal state:
+ - redcap_uri
+ - token
+ - redcap_server
+ - redcap_support_receiver_email
+ - redcap_support_sender_email
+ - smtp_host_for_outbound_mail
+ - smtp_port_for_outbound_mail
+ - sender_email (if parameter send_email = Y)
+ - receiver_email (if parameter send_email = Y)
+
+The program will terminate if they are missing or do not have a value in setings.ini. A message about this will be displayed to the user before the program terminates. The same is also written to the log file.
+If the program is executing in dry state then the above parameters are optional
+
+Following parameters are required if we run redi with -e command line argument:
+ - emr_sftp_server_hostname
+ - emr_sftp_server_username
+ - emr_sftp_server_password
+ - emr_sftp_project_name
+ - emr_data_file
+ - emr_log_file
+
+These parameters are essential for establishing connection with the sftp server to obtain EMR data, so if they are missing or do not have a value in settings.ini, then the program will terminate. A message about this will be displayed to the user before the program terminates. The same is also written to the log file.
+
+Following parameters in settings.ini are optional:
+ NAME                   DEFAULT VALUE
+ system_log_file        redi_log/redi.log
+ report_file_path       report.xml
+ input_date_format      %Y-%m-%d %H:%M:%S
+ output_date_format     %Y-%m-%d
+ report_file_path2      report.html
+ project                DEFAULT_PROJECT
+ rate_limiter           600
+ batch_warning_days     13
+
+If the above parameters are missing or do not have a value in settings.ini then the corresponding default value is used and program continues with execution. A message about default value being used is written to the log file.
+
 Testing
 -------
 1. Go to the vagrant folder and remove the old virtual machine (VM) instance:
@@ -46,7 +135,7 @@ Testing
 
     $ python <project_root_path>/test/TestSuite.py
 
-Note: One of test cases, `TestGenerateOutput`, requires a running REDCap instance. If there is no running REDCap instance details in setup.json this test case fails. This is a known issue and will be handled in future releases.
+Note: One of test cases, `TestGenerateOutput`, requires a running REDCap instance. If there is no running REDCap instance details in settings.ini this test case fails. This is a known issue and will be handled in future releases.
 
 @TODO: A sample redcap project and test dataset need to be added.
 

bin/form.py

@@ -9,8 +9,8 @@ class Form(object):
     def __init__(self, data):
         try:
             data.xpath('.')
-        except Exception as e:
-            print e
+        except Exception as data_exception:
+            print data_exception
             raise ValueError('"data" should be a valid lxml etree')
 
         self._tree = data
@@ -31,14 +31,17 @@ def __init__(self, etree_node):
         self._node = etree_node
 
     def field(self, name):
-        nodes = self._node.xpath("./field[./name = $fieldName]", fieldName=name)
+        nodes = self._node.xpath(
+            "./field[./name = $fieldName]",
+            fieldName=name)
         nodecount = len(nodes)
         if nodecount == 1:
             return Field(nodes[0])
         elif nodecount == 0:
             return None
         else:
-            raise Exception("Malformed XML: multiple fields with the name {0}".format(name))
+            raise Exception("Malformed XML: multiple fields with the name {0}".
+                            format(name))
 
     @property
     def name(self):
@@ -83,4 +86,3 @@ def value(self, value):
 
     def clear_value(self):
         self.value = ''
-