Version bump and update of Changelog (#47)

## [0.0.8] - 2019-06-29
Bringing things in line with updates to POCS for docker and panoptes-utils use.
### Changed
* Setup/Install:
    * Scripts are renamed to have `panoptes` prefix.
    * Scripts are installed as part of setup.
    * Script improments to make more robust and portable.
* Docker Updates:
    * Don't use anaconda.
* Testing:
    * Overhaul of config_server use in testing.
    * Testing config file is separated from any regular config files.
* Logging:
    * Silence some 3rd party logs.

### Bug fixes
* Serialization fixes.
    * Use our serialization everywhere (e.g. messaging)
    * Closes #panoptes/POCS/issues/818
    * Closes #panoptes/POCS/issues/103

### Added
* Serial handlers move to panoptes-utils from POCS.
* Tests and coverage.
* `improve_wcs` (moved from PIAA).
* `~utils.fits.getdata` to match other fits convenience functions, allowing for
fpack files.

.coveragerc

@@ -13,5 +13,3 @@ exclude_lines =
     pragma: no cover
 
 ignore_errors = True
-omit =
-    tests/*

CHANGELOG.md

@@ -4,17 +4,45 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.0.8] - 2019-06-29
+Bringing things in line with updates to POCS for docker and panoptes-utils use.
+### Changed
+* Setup/Install:
+    * Scripts are renamed to have `panoptes` prefix.
+    * Scripts are installed as part of setup.
+    * Script improments to make more robust and portable.
+* Docker Updates:
+    * Don't use anaconda.
+* Testing:
+    * Overhaul of config_server use in testing.
+    * Testing config file is separated from any regular config files.
+* Logging:
+    * Silence some 3rd party logs.
+
+### Bug fixes
+* Serialization fixes.
+    * Use our serialization everywhere (e.g. messaging)
+    * Closes #panoptes/POCS/issues/818
+    * Closes #panoptes/POCS/issues/103
+
+### Added
+* Serial handlers move to panoptes-utils from POCS.
+* Tests and coverage.
+* `improve_wcs` (moved from PIAA).
+* `~utils.fits.getdata` to match other fits convenience functions, allowing for
+fpack files.
+
 
 ## [0.0.7] - 2019-05-26
-## Changed
+### Changed
 * **Breaking** Changed namespace so no underscores, i.e. `from panoptes.utils import time`.
 * Docker updates:
   * Use slim python images and not anaconda on amd64.
   * Adding zsh as default shell along with some customizations.
   * Entrypoint script properly authenticates to google cloud if possible.
   * Added amd64 only build scripts.
 
-## Added
+### Added
 * Added bayer utilities. :camera:
 * Added Cloud SQL utilities. :cloud:
 

README.md

@@ -38,7 +38,7 @@ A simple config param server. Runs as a Flask microservice that delivers JSON do
 in response to requests for config key items. To start the service run:
 
 ```bash
-scripts/run_config_server.py
+run-config-server
 ```
 
 The server can be queried/set in python:
@@ -130,7 +130,7 @@ PANOPTES system. Running the Messaging Hub will set up a forwarding service that
 number of publishers and subscribers.
 
 ```bash
-scripts/run_messaging_hub.py --from-config
+panoptes-messaging-hub --from-config
 ```
 
 ## Docker

scripts/cr2_to_jpg.sh → bin/cr2-to-jpg

@@ -16,7 +16,7 @@ usage() {
   TITLE             Optional tilte to be placed on jpeg.
 
  Example:
-  scripts/cr2_to_jpg.sh /var/panoptes/images/temp.cr2 \"M42 (Orion's Nebula)\"
+  cr2-to-jpg /var/panoptes/images/temp.cr2 \"M42 (Orion's Nebula)\"
 "
 }
 
@@ -32,16 +32,23 @@ JPG="${FNAME%.cr2}.jpg"
 
 echo "Converting CR2 to ${JPG}."
 
+function command_exists {
+    # https://gist.github.com/gubatron/1eb077a1c5fcf510e8e5
+    # this should be a very portable way of checking if something is on the path
+    # usage: "if command_exists foo; then echo it exists; fi"
+  type "$1" &> /dev/null
+}
+
 # Use exiftool to extract preview if it exists
-if hash exiftool 2>/dev/null; then
+if command_exists exiftool; then
     exiftool -b -PreviewImage "${FNAME}" > "${JPG}"
 else
-    if hash dcraw 2>/dev/null; then
+    if command_exists dcraw; then
         # Convert CR2 to JPG
         dcraw -c -q 3 -a -w -H 5 -b 5 "${FNAME}" | cjpeg -quality 90 > "${JPG}"
     else
         echo "Can't find either exiftool or dcraw, cannot proceed"
-        exit
+        exit 1
     fi
 fi
 

scripts/run_config_server.py → bin/panoptes-config-server

@@ -24,6 +24,8 @@
                         help="Config file, default $PANDIR/conf_files/pocs.yaml")
     parser.add_argument('--no-save', default=False, action='store_true',
                         help='Config entries are saved automatically by default.')
+    parser.add_argument('--ignore-local', default=False, action='store_true',
+                        help='Ignore the local config files, default False. Mostly for testing.')
     parser.add_argument('--debug', default=False, action='store_true', help='Debug')
     args = parser.parse_args()
 
@@ -43,7 +45,9 @@
 
     app.config['auto_save'] = not args.no_save
     app.config['config_file'] = args.config_file
-    app.config['POCS'] = load_config(config_files=args.config_file)
+    app.config['ignore_local'] = args.ignore_local
+    app.config['POCS'] = load_config(config_files=args.config_file,
+                                     ignore_local=args.ignore_local)
     app.config['POCS_cut'] = Cut(app.config['POCS'])
 
     app.run(

scripts/solve_field.sh → bin/panoptes-solve-field

@@ -1,8 +1,8 @@
 #!/bin/bash
 
 # If SOLVE_FIELD is not set, try the default; warn if default unavailable.
-SOLVE_FIELD=${SOLVE_FIELD:-"/usr/bin/solve-field"}
-if [ ! -f $SOLVE_FIELD ]
+SOLVE_FIELD=${SOLVE_FIELD:-"solve-field"}
+if [ ! -f "$SOLVE_FIELD" ]
 then
 	echo 2>&1 ""
 	echo 2>&1 "Please install astrometry into ${PANDIR}/astrometry to enable $0"
@@ -11,8 +11,8 @@ then
 fi
 
 if [[ $# == 1 ]]; then
-    echo "Using options: --guess-scale --no-plots --downsample 3 --overwrite"
-    ${SOLVE_FIELD} --guess-scale --no-plots --downsample 3 --overwrite $1
+    echo "Using options: --guess-scale --no-plots --downsample 3 --overwrite $1"
+    "${SOLVE_FIELD}" --guess-scale --no-plots --downsample 3 --overwrite "$1"
 else
-    ${SOLVE_FIELD} $@
+    "${SOLVE_FIELD}" "$@"
 fi

conftest.py

@@ -11,10 +11,17 @@
 import pytest
 import subprocess
 import time
+import shutil
+from multiprocessing import Process
+
+from scalpl import Cut
 
 from panoptes.utils.database import PanDB
+from panoptes.utils.config import load_config
 from panoptes.utils.logger import get_root_logger
 from panoptes.utils.messaging import PanMessaging
+from panoptes.utils.config.client import set_config
+from panoptes.utils.config.server import app
 
 # Global variable set to a bool by can_connect_to_mongo().
 _can_connect_to_mongo = None
@@ -69,7 +76,6 @@ def pytest_runtest_logstart(nodeid, location):
         logger.critical('')
         logger.critical('##########' * 8)
         logger.critical('     START TEST {}', nodeid)
-        logger.critical('')
     except Exception:
         pass
 
@@ -88,7 +94,6 @@ def pytest_runtest_logfinish(nodeid, location):
         logger = get_root_logger()
         logger.critical('')
         logger.critical('       END TEST {}', nodeid)
-        logger.critical('')
         logger.critical('##########' * 8)
     except Exception:
         pass
@@ -114,6 +119,156 @@ def pytest_runtest_logreport(report):
         pass
 
 
+@pytest.fixture(scope='session')
+def config_host():
+    return 'localhost'
+
+
+@pytest.fixture(scope='session')
+def static_config_port():
+    """Used for the session-scoped config_server where no config values
+    are expected to change during testing.
+    """
+    return '6563'
+
+
+@pytest.fixture(scope='module')
+def config_port():
+    """Used for the function-scoped config_server when it is required to change
+    config values during testing. See `dynamic_config_server` docs below.
+    """
+    return '4861'
+
+
+@pytest.fixture(scope='session')
+def db_name():
+    return 'panoptes_testing'
+
+
+@pytest.fixture(scope='session')
+def images_dir(tmpdir_factory):
+    directory = tmpdir_factory.mktemp('images')
+    return str(directory)
+
+
+@pytest.fixture(scope='session')
+def config_path():
+    return os.path.join(os.getenv('PANDIR'),
+                        'panoptes-utils',
+                        'panoptes',
+                        'tests',
+                        'panoptes_utils_testing.yaml'
+                        )
+
+
+@pytest.fixture(scope='session')
+def config_server_args(config_path):
+    loaded_config = load_config(config_files=config_path, ignore_local=True)
+    return {
+        'config_file': config_path,
+        'auto_save': False,
+        'ignore_local': True,
+        'POCS': loaded_config,
+        'POCS_cut': Cut(loaded_config)
+    }
+
+
+@pytest.fixture(scope='session', autouse=True)
+def static_config_server(config_host, static_config_port, config_server_args, images_dir, db_name):
+
+    logger = get_root_logger()
+    logger.critical(f'Starting config_server for testing session')
+
+    def start_config_server():
+        # Load the config items into the app config.
+        for k, v in config_server_args.items():
+            app.config[k] = v
+
+        # Start the actual flask server.
+        app.run(host=config_host, port=static_config_port)
+
+    proc = Process(target=start_config_server)
+    proc.start()
+
+    logger.info(f'config_server started with PID={proc.pid}')
+
+    # Give server time to start
+    time.sleep(1)
+
+    # Adjust various config items for testing
+    unit_name = 'Generic PANOPTES Unit'
+    unit_id = 'PAN000'
+    logger.info(f'Setting testing name and unit_id to {unit_id}')
+    set_config('name', unit_name, port=static_config_port)
+    set_config('pan_id', unit_id, port=static_config_port)
+
+    logger.info(f'Setting testing database to {db_name}')
+    set_config('db.name', db_name, port=static_config_port)
+
+    fields_file = 'simulator.yaml'
+    logger.info(f'Setting testing scheduler fields_file to {fields_file}')
+    set_config('scheduler.fields_file', fields_file, port=static_config_port)
+
+    # TODO(wtgee): determine if we need separate directories for each module.
+    logger.info(f'Setting temporary image directory for testing')
+    set_config('directories.images', images_dir, port=static_config_port)
+
+    yield
+    logger.critical(f'Killing config_server started with PID={proc.pid}')
+    proc.terminate()
+
+
+@pytest.fixture(scope='function')
+def dynamic_config_server(config_host, config_port, config_server_args, images_dir, db_name):
+    """If a test requires changing the configuration we use a function-scoped testing
+    server. We only do this on tests that require it so we are not constantly starting and stopping
+    the config server unless necessary.  To use this, each test that requires it must use the
+    `dynamic_config_server` and `config_port` fixtures and must pass the `config_port` to all
+    instances that are created (propogated through PanBase).
+    """
+
+    logger = get_root_logger()
+    logger.critical(f'Starting config_server for testing function')
+
+    def start_config_server():
+        # Load the config items into the app config.
+        for k, v in config_server_args.items():
+            app.config[k] = v
+
+        # Start the actual flask server.
+        app.run(host=config_host, port=config_port)
+
+    proc = Process(target=start_config_server)
+    proc.start()
+
+    logger.info(f'config_server started with PID={proc.pid}')
+
+    # Give server time to start
+    time.sleep(1)
+
+    # Adjust various config items for testing
+    unit_name = 'Generic PANOPTES Unit'
+    unit_id = 'PAN000'
+    logger.info(f'Setting testing name and unit_id to {unit_id}')
+    set_config('name', unit_name, port=config_port)
+    set_config('pan_id', unit_id, port=config_port)
+
+    logger.info(f'Setting testing database to {db_name}')
+    set_config('db.name', db_name, port=config_port)
+
+    fields_file = 'simulator.yaml'
+    logger.info(f'Setting testing scheduler fields_file to {fields_file}')
+    set_config('scheduler.fields_file', fields_file, port=config_port)
+
+    # TODO(wtgee): determine if we need separate directories for each module.
+    logger.info(f'Setting temporary image directory for testing')
+    set_config('directories.images', images_dir, port=config_port)
+
+    yield
+    logger.critical(f'Killing config_server started with PID={proc.pid}')
+    proc.terminate()
+
+
 @pytest.fixture
 def temp_file():
     temp_file = 'temp'
@@ -214,7 +369,8 @@ def messaging_ports():
 
 @pytest.fixture(scope='function')
 def message_forwarder(messaging_ports):
-    cmd = os.path.join(os.getenv('PANDIR'), 'panoptes-utils', 'scripts', 'run_messaging_hub.py')
+    cmd = shutil.which('panoptes-messaging-hub')
+    assert cmd is not None
     args = [cmd]
     # Note that the other programs using these port pairs consider
     # them to be pub and sub, in that order, but the forwarder sees things

docker/Dockerfile

@@ -29,7 +29,7 @@ RUN mkdir -p ${POCS} && \
     apt-get install -y --no-install-recommends \
         wget curl bzip2 ca-certificates pkg-config zsh git fonts-lato nano \
         astrometry.net sextractor dcraw exiftool libcfitsio-dev libcfitsio-bin imagemagick \
-        libcfitsio-dev libfreetype6-dev libpng-dev libpq-dev && \
+        libcfitsio-dev libfreetype6-dev libpng-dev libpq-dev ack && \
     # Oh My ZSH. :)
     chsh -s /bin/zsh && \
     sh -c "$(wget https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh -O -)" && \

docker/Dockerfile.utils.amd64

@@ -17,10 +17,9 @@ RUN git clone https://github.com/denysdovhan/spaceship-prompt.git "$ZSH_CUSTOM/t
     pip install --no-cache-dir ".[all]" && \
     # Download astrometry.net files
     python panoptes/utils/data.py --wide-field --narrow-field
-    # TODO add cron job for IERS data download
 
 
 # Comes from base image - hard-coded for now ☹.
-ENTRYPOINT ["/var/panoptes/panoptes-utils/docker/entrypoint.sh"]
+ENTRYPOINT ["/bin/sh", "/var/panoptes/panoptes-utils/docker/entrypoint.sh"]
 
 CMD ["/bin/zsh"]

docker/Dockerfile.utils.rpi

@@ -46,10 +46,9 @@ FROM base-image
 RUN cd ${PANDIR}/panoptes-utils && \
     /opt/conda/envs/${CONDA_ENV}/bin/pip install --no-cache-dir -e ".[all]" && \
     # Download astrometry.net files
-    # TODO add cron job for IERS data download
     /opt/conda/envs/${CONDA_ENV}/bin/python panoptes/utils/data.py --wide-field --narrow-field
 
-ENTRYPOINT ["/var/panoptes/panoptes-utils/docker/entrypoint.sh"]
+ENTRYPOINT ["/bin/sh", "/var/panoptes/panoptes-utils/docker/entrypoint.sh"]
 
 CMD ["/bin/zsh"]