v0.7.0

This is the release notes of v0.7.0. See here for the complete list of solved issues and merged PRs.

This release note only covers the difference from v0.7.0rc2; for all highlights and changes, please refer to the release notes of the pre-releases:

alpha1
alpha2
alpha3
alpha4
alpha5
alpha6
alpha7
alpha8
beta1
beta2
rc1
rc2

Changes that break compatibility

v0.7.0 has unified local and distributed execution layer, local thread-based scheduling has been removed, instead, the unified runtime is based on multiprocess-based scheduling which could get rid of infamous GIL problem .

Thus, for local usage, please new a local default session via:

import mars

mars.new_session()  # create a default local session

If not doing so, it will be initialized once in the background, however, keep in mind that the initialization of multiprocess scheduling consumes more time compared to multithread one.

We tried our best to keep other compatibilities, if you find any incompatible place, please open an issue to reach out to us.

Highlights

v0.7.0 implements a unified execution layer, all deployment including bare metal, Kubernetes, Ray as well as Yarn shares the same fundamental components. This unified execution layer optimized many aspects compare to the old one including:

• Better serialization based on pickle5 protocol, which is 5-7x faster than old version.
• Completely rewritten execution layer which has better performance, even 20%-50% faster than the old version on a laptop.
• Based on multiprocess scheduling which avoids infamous GIL issue.
• Mars on Ray is way more better due to the reason that Ray actor is leveraged to build the Ray backend of Oscar which is a lightweight actor framework that is the fundamental part of the entire execution layer.
• GPU can be supported more better with the new architecture.

New Features

• Tensor
  • Add partial support of bessel functions (#2274, thanks @JuntaoMa!)
  • Implements mars.tensor.in1d (#2301)
• Learn
  • Implements mars.learn.utils.multiclass.unique_label (#2300)
• Services
  • Add get_storage_level_info api (#2242)
  • Add API to fetch tileable graph as JSON (#2271, thanks @RandomY-2!)
  • Enable running on GPU for oscar (#2306)
• Others
  • Add support for seek method in memory cases (#2264)

Enhancements

• Add support for stateless actors (#2220)
• Add status filters for Cluster service (#2221)
• Pass logging config file name into sub pools (#2225)
• Support choosing aggregation algorithm at runtime (#2226)
• Add method to session to get web endpoint (#2238)
• Use Kubernetes Service to discover Mars Supervisors (#2240)
• Ensure range index incremental for data source op like md.read_csv (#2244)
• Record mapper meta for shuffle task (#2255)
• Support data dependency for run_script (#2256)
• Refine oscar debugging (#2261)
• Support fetch_log for web session (#2262)
• Allow turning off actor killing (#2277)
• Use batch method to reduce transferring cost for shuffle tasks (#2279)
• Assign bands given devices of subtasks (#2278)
• Add bind method to facilitate extracting batch args (#2281)
• Reduce memory estimation for specific operands (#2285)

Bug fixes

• Fix NoDataToSpill when multiple storage quota requests happen simultaneously (#2223)
• Stop using thread local to store default session (#2243)
• Fix service errors in Windows (#2247)

Documentation

• Doc refinement for Oscar (#2291)
• Add docs for batch methods (#2298)

Installation

• Merge default & distributed requirements (#2270)

Tests

• Add separate check pipeline (#2302)

v0.7.0rc2

This is the release notes of v0.7.0rc2. See here for the complete list of solved issues and merged PRs.

New Features

• Services
  • Support setting task progress in context (#2192)
• Web
  • Implement essential APIs for Web (#2181)
  • Use React framework to rewrite Mars UI (#2135)
• Deploy
  • Cluster config support third party modules (#2171)

Enhancements

• Use aiohttp to handle web requests (#2183)
• Add stop methods for all services (#2194)
• Move fetch method from StorageManagerActor to StorageHandlerActor (#2196)
• Isolate client and cluster in a separated event loop and thread (#2168)

Bug fixes

• Fix unexpected NaNs in groupby-agg (#2177)
• Fix groupby on indexes with duplicate items (#2186)
• Fix starting multiple workers in shared file system (#2189)
• Fix import error in master branch (#2190)
• [Ray] ray two way hang-detectable channel (#2170)
• Fix compatibility issue for pandas 1.3 (#2197)
• Fix deserializing task errors on web clients (#2199)
• Make window function under old pandas versions work (#2204)
• Fix concatenating row chunks with MultiIndex (#2205)
• Fix merge_index_value when index_values come from multi range indexes (#2207)

v0.6.11

This is the release notes of v0.6.11. See here for the complete list of solved issues and merged PRs.

Bug fixes

• Fix unexpected NaNs in groupby-agg (#2178)
• Fix groupby on indexes with duplicate items (#2187)
• Fix compatibility issue for pandas 1.3 (#2202)
• Fix merge_index_value when index_values come from multi range indexes (#2208)

v0.7.0rc1

This is the release notes of v0.7.0rc1. See here for the complete list of solved issues and merged PRs.

New Features

• DataFrame
  • Implement {DataFrame, Series}.empty (#2155, thanks @Fernadoo!)
• Services
  • Add spill support for oscar (#2160)

Enhancements

• Migrate YARN support for Oscar (#2152)
• Add version API on cluster (#2153)
• Make sure slot usages updated when some task ends (#2161)
• Add debug options for oscar (#2164)
• Add fault injection subtask processor for tests (#2163)
• Collects available ports before running tensorflow scripts (#2169)
• Support stopping tasks for interactive execution (#2154)
• Add cycle detection for oscar debug mode (#2167)
• Update year of license (#2172)
• Support configuring overriding fields (#2173)

Bug fixes

• Fix race condition when creating actors with IdleLabel strategy in a parallel way (#2147)
• Fix shuffle task on oscar (#2146)

Tests

• Use a setup.cfg to gather configurations (#2150)

v0.6.10

This is the release notes of v0.6.10. See here for the complete list of solved issues and merged PRs.

New Features

• Implement {DataFrame, Series}.empty (#2174, thanks @Fernadoo!)

Bug fixes

• Fix _get_ports_from_netstat hang (#2174)

v0.7.0b2

This is the release notes of v0.7.0b2. See here for the complete list of solved issues and merged PRs.

Changes that break compatibility

• From v0.7.0b2 on, staled threading-based scheduler as well as distributed scheduler based on Mars actor 1.0 have been removed, thus clients with older versions are completely incompatible.

Highlights

• Unified scheduling based on Oscar which is Mars actor 2.0 is ready for tests.

New Features

• DataFrame
  • Add add_prefix support (#2132, thanks @aeinrw!)
• Services
  • Services web handler and api (#2102)
  • Implements lifecycle service (#2117)
  • Add initial implementation of scheduling service (#2111)
• Deloy
  • Add command line support for Oscar deployment (#2131)
• Ray
  • [Ray] ray oscar deploy (#2089)

Enhancements

• Hold data ref in DataManager (#2090)
• Enabling iterative tiling etc support for task service (#2097)
• Enable optimization for task service (#2098)
• Implement last_idle_time API (#2099)
• Integrate mars object check for session (#2103)
• Make Mars pools compatible with Python 3.6 (#2110)
• [ray] optimize ray deploy speed (#2118)
• Implement RESTful web API (#2120)
• [ray] Support supervisor exclusive node option (#2121)
• Add asyncio task timeout debugger (#2127)
• Add transfer support for storage service (#2100)
• [ray] supervisor support sub pool (#2128)
• Configure azure pipelines job timeout (#2139)
• Allow overriding service config files (#2140)

Bug fixes

• Fix distributed make_blobs and column pruning in read_sql (#2092)
• Fix result error when yield after exceptions (#2096)
• Wrap sync method in session so that they will be running in threads (#2109)
• Fix pool cases and shared memory cases in Windows (#2114)
• Fix _get_ports_from_netstat hang (#2116)
• Fix unpickle mars config error (#2130)
• Filter pipeline jobs by branch (#2138)

Tests

• Fix coverage result on SubActorPool (#2095)
• [Ray] Support ray subprocess covarage (#2101)
• Run operand tests in Azure Pipelines (#2137)
• Migrate tensor/dataframe/learn tests to oscar (#2106)

v0.6.9

This is the release notes of v0.6.9. See here for the complete list of solved issues and merged PRs.

New Features

• Add add_prefix support (#2133, thanks @aeinrw!)

Bug fixes

• Fix distributed make_blobs and column pruning in read_sql (#2093)

Tests

• Remove tokens for codecov on 0.6 (#2141)

v0.7.0b1

This is the release notes of v0.7.0b1. See here for the complete list of solved issues and merged PRs.

New Features

• Tensor
  • Implements mt.block (#2069, thanks @fyrestone!)
• Project Galois
  • Initial task service support (#2084)

Enhancements

• Apply new serialization to operand and Mars objects (#2075)
• Make import of LightGBM and XGBoost lazy (#2083)
• Use chunk index as default shuffle key (#2086)
• Project Galois
  • Oscar actor pool on ray (#2063, thanks @chaokunyang!)

Bug fixes

• Use a ref count to make delete works when multiple workers connect to the same vineyardd (#2077)

Tests

• Allow cancelling flows automatically (#2082)

v0.6.8

This is the release notes of v0.6.8. See here for the complete list of solved issues and merged PRs.

New Features

• Tensor
  • Implements mt.block (#2080, thanks @fyrestone!)

Enhancements

• Make import of LightGBM and XGBoost lazy (#2085)

Bug fixes

• Fix mt.unique on empty arrays (#2061)
• Use a ref count to make delete works when multiple workers connect to the same vineyard (#2078)
• Backport of bug fixes discovered in Galois serialization refactor (#2081)

Tests

• Fix batch get / delete object id and stop launching plasma when working on vineyard (#2071)

v0.7.0a8

This is the release notes of v0.7.0a8. See here for the complete list of solved issues and merged PRs.

New Features

• Tensor
  • Implement mt.insert and mt.delete (#2039)

Project Galois

• Oscar
  • [Ray backend] setup_cluster use placement group (#2041)
  • Fix cancelling actor promises (#2032)
  • Fix __pre_destroy__ not called when actors are in main pool (#2045)
  • [Ray] Fix ray cluster_utils import (#2054)
  • Fix main pool that creating servers only when sub pools finished creation (#2064)
• Services
  • Initialize meta service with mock support (#2034)
  • API definition for services (#2040)
  • Allow sync actor methods to be extensible (#2050)
  • Implements initial version of cluster service (#2049)
  • Enhance meta service (#2062)
  • Initial implementation for storage service (#2056)
  • Add bands to chunk meta (#2065)
  • Enrich usage experience for extensible function (#2067)
  • Add API interface to get data info (#2068)
• Serialization
  • Support cuda buffer serializations in actor communication (#2031)
  • Allow serializer to serialize recursive objects (#2058)
  • Implements serializables based on new serialization mechanism (#2051)

Enhancements

• Support reusing kubedl cluster by job name (#2035)
• Quarantine asyncio tests when measuring Cython coverage (#2070)

Bug fixes

• Fix wrong results of mt.insert (#2046)
• Fix for mt.insert when insert values is a mars tensor (#2052)
• Fix batch get / delete object id and stop launching plasma when working on vineyard (#2072)