vineyard v0.2.10

Vineyard v0.2.10 is a minor version, where,

1. a vineyard.init() and vineyard.shutdown() interface has been added to launch vineyard in eaiser way
2. a bug in dataframe builder has been fixed to ensure zero-copy and reusing chunks.

Commits

• [d492bc0]: Update dynamic hash partitioner (#506) (Weibin Zeng)
• [2b1cb09]: Update modules in client/ds and add singleton for Rust SDK (#502) (Sijie)
• [fed7981]: Reusing alreay-in-vineyard columns by inspecting blocks of dataframe. (#504) (Tao He)
• [4fc9247]: Fixes the missing header. (#500) (Tao He)
• [3f88a0b]: Fixes usage of start_vineyardd, formatting, and docs. (#499) (Tao He)
• [c20b406]: Add the pair of init() and shutdown() methods to help new users to deploy vineyardd easily (#498) (Diwen Zhu)
• [d6eb1b1]: Adds start command to vineyard-ctl (#497) (Rohan Gupta)
• [f09c028]: Make the distributed learning example runnable on a single machine (#495) (Diwen Zhu)
• [c8dc6cb]: Airflow: unitest, and support easier launching of vineyard server. (#494) (Tao He)

vineyard v0.2.9

Commits

• [75f0877]: Upgrade to vineyard v0.2.9 for bug fixes. (Tao He)
• [0e242c8]: Add a vineyard exception class in Python. (#493) (Tao He)
• [5e2125a]: Enable constexpr PRETTY_FUNCTION only for gcc >= 9.1.0. (#492) (Tao He)
• [2062fe3]: Adds memory_status option to query command in vineyard-ctl (#476) (Rohan Gupta)
• [f941ef7]: Update Rust SDK object_meta module (#478) (Sijie)
• [5218f24]: Upload more wheels to the tagged release. (#490) (Tao He)

vineyard v0.2.8

Vineyard v0.2.8 is a bugfix release.

Commits

• [94c5e54]: Update title markup. (Tao He)
• [534a1d8]: Add a notebook of a distributed learning example (#487) (Diwen Zhu)
• [b50106e]: Typename: compatible with higher version of GCC. (#486) (Tao He)
• [d88d31e]: Support injecting extra metadata in existing objects via a shallow copy. (#482) (Tao He)
• [d2d166a]: Support adding metadata with __setitem__ operator on ObjectMeta. (#480) (Tao He)
• [0856074]: Vineyard IO: raise exceptions when encounter errors in writers (in its join). (#481) (Tao He)
• [bf41f8b]: Fixes a crash when adding invalid metadata. (#479) (Tao He)
• [ae9afba]: Fixes the use-after-free bug for string-view in the oid array. (#475) (Tao He)
• [c4d8b90]: Add tree command in vineyard-ctl to output object lineage in tree-like style (#474) (Rohan Gupta)
• [3a8fb19]: Revisit the implementation of typename: more customizable and stable. (#472) (Tao He)
• [9f6d1ae]: Adds debug command to vineyard-ctl (#473) (Rohan Gupta)
• [9aed745]: Git attributes: assoicate .vineyard-mod to C++. (Tao He)
• [10a730e]: Ray dataset: from and to vineyard. (#470) (Tao He)
• [25f15d1]: Move the newly added item to the end of the enum. (#468) (Tao He)
• [7e1c672]: Adds DeepCopy function to vineyard and Python SDK and adds it to vineyard-ctl (#467) (Rohan Gupta)
• [f675381]: Update init.py to import VineyardXCom (#466) (andtheysay)
• [28856ae]: Add data structure date32 and date64 in C++ SDK (#439) (beiluo)
• [c0cf8ba]: Add more metrics for prometheus (#465) (Jiang Shanshan)
• [3de4b98]: Use a lock to protect the mutable write buffer queue. (#464) (Tao He)
• [3f7257a]: Add write table method to i_io_adaptor (#462) (Siyuan Zhang)
• [625b7d3]: Added MXNet Builders, Resolvers and Tests (#452) (Chaitravi Chalke)
• [3d842f4]: Vineyard tests: format & lint. (#461) (Tao He)
• [9ad80c6]: Add connect put/get/drop_name functions and tests for Rust. (#460) (Sijie)
• [73074f5]: Update Dask Doc (#458) (Diwen Zhu)
• [4b70931]: Feed a hand-written copy-ctor and copy-assignment to ObjectMeta. (#456) (Tao He)

vineyard v0.2.7

Vineyard v0.2.7 brings many new features into experimental stage, including

• Airflow Integration with Vineyard: vineyard serves as a XCom backend for airflow to support the tasks in user-defined workflow shares large volume of intermediate data using vineyard, and enables more possibilities for orchestrating bigdata analytical workloads with Airflow. To try Airflow with vineyard, please refer to https://v6d.io/notes/airflow.html

• Vineyard brings Dask integration in this release, data in vineyard can be consumed and produced by dask procedures, and can be further shared by other compute engines that works on Vineyard. For a end-to-end demo, please refer to https://v6d.io/notes/dask.html

• Vineyard now can serve as the data source for ML frameworks, e.g., tensorflow, pytorch and xgboost. (thanks @chaitravi-ce )

• Vineyard Go SDK (thanks @linlih) and Java SDK (thanks @zhanglei1949) comes into a very preliminary stage.

Vineyard v0.2.7 will the last release of v0.2.x series and v0.3.0 will come next week!

Commits

• [491fbbc]: Upgrade vineyard to v0.2.7, and add a version number config in headers. (Tao He)
• [cc1d2af]: Fixes a bug in read response and native lib output directory, and adapt to Vineyard Java Fragment. (#453) (Tao He)
• [7050e63]: Fix a tiny bug in tensorflow resolver (#451) (Diwen Zhu)
• [00c5bfe]: Fixes the RPATH problem on Mac. (#450) (Tao He)
• [bbe1c69]: Add doc for a dask tensorflow example (#440) (Diwen Zhu)
• [be6f4ad]: Bump junit from 4.11 to 4.13.1 in /java/modules (#447) (dependabot[bot])
• [1b36571]: Add a vineyard::Blob -> ArrowBuf resolver. (#446) (Tao He)
• [8c6357a]: The objectmeta interface, and packaging script for FFI resolver on metadata (#441) (Tao He)
• [c2ccc65]: Implements a static constructor to get metadata from unsafe sources. (#445) (Tao He)
• [7d99778]: Ensure mmap in the same order with send_fd in server. (#444) (Tao He)
• [84f2164]: Drop unused rust code from the /src directory. (#443) (Tao He)
• [a558ed1]: Catch JSON error as well when reading the RPC request from client. (#442) (Tao He)
• [65482c5]: Memory allocator tests and benchmarks. (#438) (Pei Li)
• [1ec9f19]: Add ipc, proposal and rpc for Rust (#437) (Sijie)
• [2724ca7]: Add doc and quick fix to xgboost (#436) (Diwen Zhu)
• [46d820d]: Fix invalid step (#435) (Siyuan Zhang)
• [5830e1a]: Add CI for Rust (#434) (Siyuan Zhang)
• [d4db525]: Use unique_ptr to allow better flexibility for FFI interfaces. (#433) (Tao He)
• [c964ae3]: Add Go SDK put/get/drop name api implementation (#432) (linlih)
• [aa9bd17]: Add a troubleshooting section about the "permission error". (#431) (Tao He)
• [00278cf]: Added Global Resolvers and Tests for PyTorch and Documentation (#416) (Chaitravi Chalke)
• [3add0e7]: Add docs and example dags for airflow integration. (#429) (Tao He)
• [d34a4ca]: Bug Fix: property_graph_util::VertexDataColumn subtraction between shared_ptr and ptrdiff_t (#428) (Zhang Lei)
• [364cbfa]: Refine the resolver context implementation, and refactor duplicated code (#425) (Tao He)
• [e7ed7c3]: fix template type in vertexDataColum (#424) (Zhang Lei)
• [38d6957]: Lint: reduce code duplication, remove unused imports, isort, etc. (#421) (Tao He)
• [794ce15]: Add a command line arguments to control whether printing the metrics. (#420) (Tao He)
• [793b5df]: Add missing init.py to dask module otherwise it is an empty package. (#417) (Tao He)
• [05820a5]: Adds migrate command to vineyard-ctl (#419) (Rohan Gupta)
• [a5d3c5c]: [MRG]🎉 init prometheus setting (#418) (Jiang Shanshan)
• [06fe9df]: Add Dask integration: vineyard as data sink (#415) (Diwen Zhu)
• [d800c4f]: Airflow: support running among distributed celery workers. (#414) (Tao He)
• [5b16173]: Add Dask integration: vineyard as data source (#409) (Diwen Zhu)
• [262c484]: Added Tests for Tensorflow (#410) (Chaitravi Chalke)
• [22055c2]: Associate the client object to resolved value as well when possible. (#413) (Tao He)
• [dca1b69]: Vineyard XCom backend for apache-airflow. (#411) (Tao He)
• [f729e93]: Upgrade etcd-cpp-apiv3 to make it compatible with cmake less than 3.10. (Tao He)
• [b8efbb6]: Added Global Resolvers for Tensorflow and Local Builders and Resolvers for PyTorch (#395) (Chaitravi Chalke)
• [fb448e3]: Allow the IPC socket connectable by other users. (#407) (Tao He)
• [44bf8f4]: Adds documentation for vineyard-ctl (#405) (Rohan Gupta)
• [b1268cf]: Adds new commands, arguments and examples to vineyard-ctl (#401) (Rohan Gupta)
• [e03cf51]: Run xgboost tests use a scoped fixture, rather than populating the global ctx (#403) (Tao He)
• [b448a9f]: Add docs for the resolver/builder/driver context. (Tao He)
• [5505cc7]: Doc chapter for Vineyard ctl. (#402) (Tao He)
• [dd987b1]: Tidy up contrib/ml/xgboost with new resolver context (#400) (Diwen Zhu)
• [ad03562]: Contextual resolver/builder/driver for flexible nested dynamic registry. (#399) (Tao He)
• [4b4f9f8]: Enable contrib-ml test, and install xgboost deps. (#396) (Tao He)
• [26b40d4]: Add a doc chapter for ML integration. (#397) (Tao He)
• [7e0d44a]: Add xgboost integration and test (#394) (Diwen Zhu)
• [71e326b]: Add go sdk rpc/ipc client connect and close api implementation (#392) (linlih)
• [7f6ec86]: Fixes write bytes and formatting of objectmapper. (#390) (Tao He)
• [[e5978cc](https://github.com/alibaba/libviney...

vineyard v0.2.6

Vineyard: bump version to v0.2.6.

Signed-off-by: Tao He <[email protected]>

vineyard v0.2.5

Vineyard v0.2.5 includes a lot of performance improvements.

Commits

• [789b19f]: Fixes the doc: we don't have vineyard.{array,dataframe}. (#355) (Tao He)
• [84a039d]: Don't do iterate & delete-by-key at the same time. (#354) (Tao He)
• [d921873]: Support unicode path in io adaptor (#353) (Siyuan Zhang)
• [5162953]: Fixes the buffer overflow error in get_demangled_name. (#351) (Tao He)
• [c1dcc64]: Graceful clean-up on bulk store.(#333) (#350) (Pei Li)
• [d739501]: Improve the implementation of create_metadata. (#347) (Tao He)
• [889e198]: Resolve a FIXME, avoid redundant copy when building contents to blob. (#349) (Tao He)
• [490b8f2]: Suggest use make -j$(nproc) to users. (#348) (Tao He)
• [5f6505b]: Enhance the tensor builder to return a customized subclass. (#346) (Tao He)
• [d420518]: Put implementation for platform-specific APIs into a common env.cc. (#344) (Tao He)
• [ebad256]: Initialize the "handled_rev_" with a proper value. (#343) (Tao He)
• [ac4df16]: Fixes the state binding in metadata.items(). (#342) (Tao He)
• [b60ab12]: Revise the install instruction for apache-arrow. (#340) (Tao He)
• [88ee1d1]: Enhance the support for migration+deletion. (#338) (Tao He)
• [e349cbe]: [FEATURE-316] Condition to check maximum shared memory size (#322) (Rohan Gupta)
• [5862692]: Fixs issues in associate a backtrace to erroneous status.(#326) (#335) (Pei Li)
• [ab1ce67]: Install shared-memory38 as deps for python<=3.7 as well. (#334) (Tao He)
• [015a457]: Revise the issue/pull-request template on Github. (#332) (Tao He)
• [b93b342]: Improved Documentation and Added Patch (#324) (Chaitravi Chalke)
• [6e9c9cc]: Use a std::mutex to protect registered_callbacks_. (#331) (Tao He)
• [e6aa468]: Adding timing information when running pytest. (#328) (Tao He)
• [9091175]: Adding the shared-memory38 dependency. (#329) (Tao He)
• [c0ca234]: Improve the efficiency of vineyardd by avoid establishing new watches in requestUpdates (#313) (Diwen Zhu)

vineyard v0.2.4

Vineyard v0.2.4 refactor the CI settings for better development experience, other enhancement and fixes are highlighted as follows:

• Vineyard v0.2.4 adds a SyncMetadata method on the client to issue a sync request immediately to fetch updates from etcd.
• Vineyard v0.2.4 has finished a lot improvements on the Python SDK, especially support for bytes, memoryview and NumPy types.
• An optional command-line arguments -rpc/-norpc is added to vineyardd to conditionally disable the RPC service.
• Vineyard v0.2.4 enhances the start_vineyardd a lot to make launching vineyard server easier for fast testing.
• Vineyard v0.2.4 has better support for mars, see also mars#2162.

Commits

• [1e224ac]: Enhancement on vineyard resolver/builder for better support for native types. (#319) (Tao He)
• [dc42e51]: Fixes issues in deploying vineyardd using python interface. (#320) (Tao He)
• [040e562]: Add a command-line option to vineyardd for disabling the RPC service. (#318) (Tao He)
• [f36224b]: Update MAINTAINERS to add an explicit email column. (Tao He)
• [1d5e1c0]: Fixes the inconsistent description in docs. (#314) (Tao He)
• [28f56a9]: Reduce the CI loads to save time and avoid been disabled. (Tao He)
• [1b616c5]: Object builder for tuple and pair: accept an objectid as well. (#312) (Tao He)
• [be69d5a]: Add a "SyncMetaData" method on client to sync remote metadata in etcd immediately. (#310) (Tao He)
• [3f8988b]: Bump up the etcd-cpp-apiv3 version for a bugfix in keepalive. (Tao He)
• [0d7d888]: Test runner: put etcd data dir to /tmp when running on MacOS. (#308) (Tao He)
• [8a9b573]: Fix incorrect length of offset array (#306) (Siyuan Zhang)
• [4a9fd2b]: Hotfix for #295 (#300) (Siyuan Zhang)
• [3c91911]: Remove mapping generation when validate (#299) (Siyuan Zhang)
• [7fff7d5]: Add summer of code redirection. (Tao He)
• [0b04596]: Add google analytical links to docs. (Tao He)

vineyard v0.2.3

Commits

• [5cfaa6f]: Enhancement and bugfix of fragment related stuffs. (#295) (Siyuan Zhang)
• [98a0b7d]: Fixes data-race in delete (local) metadata, and start heartbeat when previous round finished. (#293) (Tao He)
• [8b6d937]: Implements "client.AllocatedSize" API. (#292) (Tao He)
• [1ef5807]: Enhance the helm charts implementation. (#288) (Tao He)
• [777aad8]: Fixes for 254db77 and 2c3e3f8. (Tao He)
• [254db77]: Fixes copy artifacts in CI script. (Tao He)
• [2c3e3f8]: Fixes vineyardd location in Docker image. (Tao He)
• [daa78ea]: Use vineyard's logo as favicon.ico. (Tao He)
• [8e73f6f]: Fixes etcd image tag, as "latest" doesn't exist anymore. (Tao He)
• [f9b2f8b]: Docutils 0.17.x has problem with sphinx_rtd_theme. (Tao He)
• [b5e906a]: Add CNCF logo in svg format. (Tao He)
• [f0a350a]: Update README.rst (Jingbo Xu)
• [5b5c955]: Refine roadmaps, and we don't requires CLA anymore, rather, we ask for DCO. (Tao He)

vineyard latest: pre-release for latest vineyard

Commits

• 4a9fd2b: Hotfix for #295 (#300) (Siyuan Zhang) #300

vineyard v0.2.2

Vineyard 0.2.2 is the first pre-release after vineyard joined CNCF as a sandbox project. In this release,

• A prototype for the kubernetes scheduler plugin that aware the locality of vineyard objects and jobs that requires those objects has been published.
• The segmentation fault bug in stacktrace has been fixed.
• The etcd-cpp-apiv3 has been updated to the latest release as several concurrent bugs related to "lock" action on etcd has been fixed.

Commits

• [01e1e50]: A helper to provide a bytes interface for r/w on Python object. (#286) (Tao He)
• [90b9e7d]: CI: hotfix for docker-push error. (Tao He)
• [b549f9a]: Implements vineyard-operator to schedule jobs based on data locality. (#223) (Tao He)
• [bf019cf]: Use a customized version of tmate action to make debug easier. (#285) (Tao He)
• [cf1fbf5]: Add tiny logo for CNCF and sandbox. (Tao He)
• [457fc49]: Use sandbox logo in README. (Tao He)
• [3c9133a]: Enable CodeQL test on CI. (#284) (Tao He)
• [5be0d27]: Upgrade etcd-cpp-apiv3, and add CNCF badge in README. (Tao He)
• [ad42ad2]: Fixes codacy errors. (#283) (Tao He)
• [3b9ae8c]: Resize of CNCF footer. (Tao He)
• [11efa26]: FOSSA and CII Best Practices badage. (Tao He)
• [08dc673]: Add CNCF logo into repo. (Tao He)
• [3e36ae6]: Add footer configuration and site meta for SEO. (Tao He)
• [0c28121]: Move to v6d-io, and add CNCF footer. (Tao He)
• [5d2ecf4]: Update vineyard logo. (Tao He)
• [73a5095]: Add more concurrent tests on meta service. (#282) (Tao He)
• [4dd2702]: Put the "License" part to the end of the page. (Tao He)
• [96c06be]: Update the FAQ section in documentation (#281) (Diwen Zhu)
• [3e1b6e0]: Fixes dots. (Tao He)
• [9f0f5ae]: Include a FAQ page in documentation. (Tao He)
• [61904dd]: Force install clang-format. (Tao He)
• [fc74d7c]: Migrate slack to CNCF. (Tao He)
• [5733c01]: Update the arrow web link (#278) (HouliangQi)
• [2415bb1]: Update vineyard logo. (Tao He)
• [f5164f5]: Make sure the vineyard server is gracefully stopped. (#271) (Tao He)
• [64a99b1]: Fixes stacktrace by avoiding use TLS sections. (#269) (Tao He)
• [cd4d222]: Add column to vertex: support both arrow::Array and (aligned)arrow::ChunkedArray. (#268) (Tao He)
• [e902c3c]: Fixes a bug in LocalIOAaptor: the operator & doesn't has specified sequential order. (#267) (Tao He)
• [30d3e14]: Ensure vineyard could be killed using ctrl-c. (#266) (Tao He)
• [27ce702]: Enabling treating vineyard as a submodule for an external project. (Tao He)
• [e77b692]: Keep backwards compatiblity. (Tao He)
• [0d1126c]: Rename EXPERIMENTAL_ON to NETWORKX (#265) (Weibin Zeng)