Skip to content

Scale 7.0.1
Compare
Choose a tag to compare
@emimaesmith emimaesmith released this 28 Oct 17:44
· 64 commits to master since this release
7.0.1
5b8feaf

Seed Job Support

Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.

Dependencies

  • DC/OS 1.10+
  • Docker 1.13+
  • ElasticSearch 2.4
  • PostgreSQL 9.4+, PostGIS 2.0+
  • Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
  • Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Breaking Changes

The v5 REST API has been deprecated and replaced by the v6 REST API. See the REST API documentation for all of the details on the v6 REST API. The following list details some of the major changes from v5 to v6:

  • The /v5/jobs/executions/, /v5/job-executions/ and /v5/job-executions/{id}/ APIs are deprecated and have been replaced by /v6/jobs/{job-id}/executions/ and /v6/jobs/{job-id}/executions/{exe-num}/.
  • The /v5/recipes/{id}/ API has been replaced by a v6 version. The new v6 version has a different response: the "inputs" section is removed, the "data" section is renamed to "input", and the "recipe_type" section will only contain "id", "name", "version", "title", and "description". The /v5/queue/new-recipe/ and /v5/recipes/{id}/reprocess/ APIs are also replaced with new v6 versions with the same changes in the response.
  • The /v5/jobs/, /v5/jobs/{id}/, /v5/queue/new-job/, and /v5/queue/requeue-jobs/ APIs have been replaced with new v6 versions.
  • The v5 sources, products, and import/export REST APIs have been deprecated. There are no v6 versions of these APIs.

Deprecated

  • Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.

Known Issues

N/A

New Features

N/A

Enhancements/Updates

  • Providing better return indicating stale scheduler (#1770)
  • Removed transfer_suffix from required for directory strikes (#1768)
  • Protecting AWS Credentials (#1763)
  • Updated data filter condition documentation (#1760)
  • Added bulk dataset member creation (#1744)
  • Dumped the ScaleFile JSON to INPUT_METADATA JSON file (#1743)
  • Finished adding batches capability (#1722)
  • Updated job-types REST API endpoint consistency (#1721)
  • Added job type deprecation warning (#1718)
  • Optimized the endpoint response creation (#1717)
  • Pinning recipe types on ID instead of name (#1711)
  • Adding recipe validation warning if the name is not an existing recipe (#1700)
  • Cleaned up old nodes in Scale (#1627)
  • Added message queue depth to the Scale scheduler status (#1576)
  • Added log system, rabbit queue, and vault health to the status API (#1532)
  • Parsed input file meta-data (#1289)
  • Added data set capability (#1067)

Bug Fixes

  • Corrected Recipe Dependency Checking (#1808)
  • Scale was not handling optional output to input connections properly (#1806)
  • Updated the CORS Headers for Scale APIs (#1804)
  • Returning the priority with job details in API (#1798)
  • Fixed the job type manifest migration (#1796)
  • Passed dependency settings to webserver (#1789)
  • Allow auto_update on job type patch (#1785)
  • Fixed issue where all jobs were configured with the newest manifest (#1782)
  • Fixed issue with empty mesos resources response (#1778)
  • Corrected issue with the input metadata manifest JSON where the permalink URL was not formulated properly (#1775)
  • Corrected intermittent batch test failure (#1773)
  • Corrected 504 error when no dataset members are created (#1772)
  • Updated more-itertools version for python 2.7 (#1767)
  • Corrected issue where API reports incorrect STATIC_URL (#1764)
  • Corrected issue where jobs did not fail when required JSON outputs are not present (#1759)
  • Corrected issue with Scale Dockerfile (#1752)
  • Updated bootstrap deployment to pass through environment variables to the UI (#1751)
  • Corrected 500 Error from workspace validation (#1733)
  • Corrected bug where timeout is ignored (#1728)
  • Corrected deployment issue where support images overwrote the main Scale image (#1726)
  • Corrected issue with the scheduler losing connection to mesos (#1716)
  • Investigated issue with Scale File v6 API returning NULL for most fields (#1678)

Database Migrations

These are relatively quick database changes applied using Django migrations to alter tables, constraints, indexes, etc. The migrations are applied during the start up of the Scale scheduler.

Database Updates

These are long running database changes related to actual data row updates that run in the background in a Scale system task. These updates are performed while Scale is running so they don't increase down time when deploying a new Scale version.

  • N/A
Assets 2
Loading