Releases: ngageoint/scale
7.2.1
7.2.0
7.2.0-rc2: Sharedmem skip (#1907)
* sharedmem added to invalid_resources check * scheduler properly stops jobs with sharedmem > 0 * removed sharedmem from MIN_RESOURCE * reverted last commit and made a check for zero value sharemem resources * reverted model to always strip sharemem resource * reverted comment
5.9.8
This release should be used instead of any previous 5.9 series release. Only build problems have been addressed since 5.9.7.
This is a patch release that allows Scale to recognize a loss of connection to the Mesos Master properly and terminate - previously scheduler process would continue running in an inactive state. Since Scale runs within Marathon, it will be able to recover when Mesos Master availability is restored.
Dependencies
- DC/OS 1.10+
- Docker 1.13+
- ElasticSearch 2.4
- PostgreSQL 9.4+, PostGIS 2.0+
- Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
- Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)
Bug Fixes
- Versioning not reflected properly in Docker images and API
- Docker images for 5.9.7 not consistent with code-base.
- Updated deployment to no longer fail due to mismatched Python 2 / 3 package
Scale 7.0.1
Seed Job Support
Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.
Dependencies
- DC/OS 1.10+
- Docker 1.13+
- ElasticSearch 2.4
- PostgreSQL 9.4+, PostGIS 2.0+
- Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
- Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)
Breaking Changes
The v5 REST API has been deprecated and replaced by the v6 REST API. See the REST API documentation for all of the details on the v6 REST API. The following list details some of the major changes from v5 to v6:
- The /v5/jobs/executions/, /v5/job-executions/ and /v5/job-executions/{id}/ APIs are deprecated and have been replaced by /v6/jobs/{job-id}/executions/ and /v6/jobs/{job-id}/executions/{exe-num}/.
- The /v5/recipes/{id}/ API has been replaced by a v6 version. The new v6 version has a different response: the "inputs" section is removed, the "data" section is renamed to "input", and the "recipe_type" section will only contain "id", "name", "version", "title", and "description". The /v5/queue/new-recipe/ and /v5/recipes/{id}/reprocess/ APIs are also replaced with new v6 versions with the same changes in the response.
- The /v5/jobs/, /v5/jobs/{id}/, /v5/queue/new-job/, and /v5/queue/requeue-jobs/ APIs have been replaced with new v6 versions.
- The v5 sources, products, and import/export REST APIs have been deprecated. There are no v6 versions of these APIs.
Deprecated
- Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.
Known Issues
N/A
New Features
N/A
Enhancements/Updates
- Providing better return indicating stale scheduler (#1770)
- Removed transfer_suffix from required for directory strikes (#1768)
- Protecting AWS Credentials (#1763)
- Updated data filter condition documentation (#1760)
- Added bulk dataset member creation (#1744)
- Dumped the ScaleFile JSON to INPUT_METADATA JSON file (#1743)
- Finished adding batches capability (#1722)
- Updated job-types REST API endpoint consistency (#1721)
- Added job type deprecation warning (#1718)
- Optimized the endpoint response creation (#1717)
- Pinning recipe types on ID instead of name (#1711)
- Adding recipe validation warning if the name is not an existing recipe (#1700)
- Cleaned up old nodes in Scale (#1627)
- Added message queue depth to the Scale scheduler status (#1576)
- Added log system, rabbit queue, and vault health to the status API (#1532)
- Parsed input file meta-data (#1289)
- Added data set capability (#1067)
Bug Fixes
- Corrected Recipe Dependency Checking (#1808)
- Scale was not handling optional output to input connections properly (#1806)
- Updated the CORS Headers for Scale APIs (#1804)
- Returning the priority with job details in API (#1798)
- Fixed the job type manifest migration (#1796)
- Passed dependency settings to webserver (#1789)
- Allow auto_update on job type patch (#1785)
- Fixed issue where all jobs were configured with the newest manifest (#1782)
- Fixed issue with empty mesos resources response (#1778)
- Corrected issue with the input metadata manifest JSON where the permalink URL was not formulated properly (#1775)
- Corrected intermittent batch test failure (#1773)
- Corrected 504 error when no dataset members are created (#1772)
- Updated more-itertools version for python 2.7 (#1767)
- Corrected issue where API reports incorrect STATIC_URL (#1764)
- Corrected issue where jobs did not fail when required JSON outputs are not present (#1759)
- Corrected issue with Scale Dockerfile (#1752)
- Updated bootstrap deployment to pass through environment variables to the UI (#1751)
- Corrected 500 Error from workspace validation (#1733)
- Corrected bug where timeout is ignored (#1728)
- Corrected deployment issue where support images overwrote the main Scale image (#1726)
- Corrected issue with the scheduler losing connection to mesos (#1716)
- Investigated issue with Scale File v6 API returning NULL for most fields (#1678)
Database Migrations
These are relatively quick database changes applied using Django migrations to alter tables, constraints, indexes, etc. The migrations are applied during the start up of the Scale scheduler.
Database Updates
These are long running database changes related to actual data row updates that run in the background in a Scale system task. These updates are performed while Scale is running so they don't increase down time when deploying a new Scale version.
- N/A
5.9.6
This is a patch release that allows Scale to recognize a loss of connection to the Mesos Master properly and terminate - previously scheduler process would continue running in an inactive state. Since Scale runs within Marathon, it will be able to recover when Mesos Master availability is restored.
Dependencies
- DC/OS 1.10+
- Docker 1.13+
- ElasticSearch 2.4
- PostgreSQL 9.4+, PostGIS 2.0+
- Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
- Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)
Bug Fixes
- #1716 - Scheduler connection to Mesos Master is lost
Scale 7.0.0
NOTE!!!!!!!
The next release needs to tell users to transition to use input file meta-data (replacing parse system) and using ingest recipes (replacing trigger system) when these replacement features are added. Also the next release should comment on message changes so admins should pause Scale and flush jobs through before upgrading Scale.
When releasing Scale v6.0.0, we need to include prominently in the release notes that legacy jobs (legacy interface in their job type revision) will not be able to be queued (they will be ignored).
Major v7.0.0 Changes
-Due to major command message changes, when upgrading to Scale v7.0.0, admins should pause Scale and flush jobs through before upgrading.
- Triggers are no longer supported. Users should update their strike configuration to include a recipe name.
- Legacy job types are no longer supported and will not be able to be queued. Job Types should be ported to the Seed interface.
Seed Job Support
Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v7.0.0, all support for legacy job types will be removed.
Dependencies
- DC/OS 1.10+
- Docker 1.17.12+
- ElasticSearch 5+
- PostgreSQL 9.4+, PostGIS 2.0+
- Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
- Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)
Breaking Changes
The v5 REST API has been deprecated and replaced by the v6/v7 REST API. See the REST API documentation for all of the details on the v6/v7 REST API. The following list details some of the major changes from v5 to v6/v7:
- The /v5/jobs/executions/, /v5/job-executions/ and /v5/job-executions/{id}/ APIs are deprecated and have been replaced by /v6/jobs/{job-id}/executions/ and /v6/jobs/{job-id}/executions/{exe-num}/.
- The /v5/recipes/{id}/ API has been replaced by a v6 version. The new v6 version has a different response: the "inputs" section is removed, the "data" section is renamed to "input", and the "recipe_type" section will only contain "id", "name", "version", "title", and "description". The /v5/queue/new-recipe/ and /v5/recipes/{id}/reprocess/ APIs are also replaced with new v6 versions with the same changes in the response.
- The /v5/jobs/, /v5/jobs/{id}/, /v5/queue/new-job/, and /v5/queue/requeue-jobs/ APIs have been replaced with new v6 versions.
- The v5 sources, products, and import/export REST APIs have been deprecated. There are no v6 versions of these APIs.
- Deprecated the "reprocess_recipes" message in favor of a create-recipes message (Issue 1172)
Deprecated
- Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant. Legacy jobs will no longer be queued (they will be ignored).
- v5 Recipe Types are now deprecated. Please transition all of your recipe types to be v6 compliant.
- Trigger Rules are now deprecated. Please transition all of your strikes to include a recipe type.
Known Issues
- Batch is not yet completely functional. This is planned to be completed by 7.1.0
New Features
- N/A
Enhancements/Updates
- Now showing job/job type max tries (#1697)
- Added is_active filtering to job type status (#1696)
- Updated the way nvidia-docker is called (#1685)
- Added an authentication toggle for testing purposes (#1672)
- Added checklist for system testing (#1669)
- Made media types not required to match (#1668)
- Added a diagnostic recipe (#1667)
- Added JSON output to result data (#1660)
- Updated job_id and exe_num index to unique (#1659)
- Removed old job_exe fields (#1658)
- Kick off recipes from ingest jobs (#1656)
- Allowed cancel of old strike jobs (#1644)
- Removed old database updater code (#1641)
- Attached conditions to optional output (#1634)
- Added default registry to silo deploy (#1631)
- Added documentation for the DATABASE_URL usage (#1617)
- Added shoudl_be_retried field to error API (#1613)
- Added retry of system errors (#1611)
- Allowed editing of recipe type is_active flag (#1608)
- Updated validation for seed manifest to match the seed cli (#1604)
- Added seed-silo to the scale deploy (#1599)
- set_recipe_input_data_v6 contained trigger references (#1591)
- Temporarily disabled batches (#1581)
- Added error when creating two errors with the same name/job-type-name combination (#1567)
- Converted configurations to v6 (#1551)
- Updated the recipe type documentation (#1540)
- Populated legacy job data (#1537)
- Added support for periods in API urls (#1536)
- Replaced logstash with fluentd (#1531)
- Updated the Job Type base serializer to include is_paused and unmet_resources (#1528)
- Added missing v6 job type endpoints (#1527)
- Added fields to recipes API for gallery view (#1526)
- Backport decline offers patch (#1525)
- Verified X-Pack protected Elasticsearch support (#1520)
- Outstanding Scale offers in mesos (#1516)
- Reset the unmet_resources flag after a while (#1511)
- Added
v7support for all API calls (#1508) - Added the job-type-names endpoint (#1503)
- Removed the v5 APIs (#1500,#1501,#1502)
- Added job type resource warnings for resources with non-supported names (#1493)
- Added resource warnings for jobs that will never be scheduled due to unfulfilled resources (#1492)
- Added support for a Cancel All button (#1483)
- Added ID to job type list REST API results (#1482)
- Added warnings displaying cleanup failures (#1445)
- Removed old recipe code (#1377)
- Added details for queued jobs (#1335)
- Disabled trigger rules in v5 (#1322)
- Removed greedy GPU scheduling logic (#1306)
- DCOS SDK integration (#1282)
- Added the Scale v6 UI into Docker images (#1270)
- Removed the trigger system (#1181)
- Removed legacy job types (#1155)
- Clear out database update (#1149)
- Changed the job execution index to be unique (#1055)
- Return all JSON fields in correct version for API called (#1042)
- Scheduler Warnings/Errors (#1032)
- Added 'Cancel All' functionality to Jobs list (#682)
- Removed job type is_privileged (#621)
- Removed the Job Type docker_params (#588)
- Updated the Meso interface install (#406)
- Django Auth Integration (#280)
- Upgraded to the latest Mesos ([#21](https://github.com/ngag...
5.9.5
This is a patch release to fix bugs found in 5.9.4
Dependencies
- DC/OS 1.10+
- Docker 1.13+
- ElasticSearch 2.4
- PostgreSQL 9.4+, PostGIS 2.0+
- Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
- Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)
Bug Fixes
5.9.4
5.9.3
This is a patch release to address several bugs: #1545, #1537, #1536, #1538, #1541
Dependencies
- DC/OS 1.10+
- Docker 1.13+
- ElasticSearch 2.4
- PostgreSQL 9.4+, PostGIS 2.0+
- Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
- Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)