vineyard v0.8.0

Vineyard v0.8.0 introduces GPU memory support, and has addressed a race condition bug that may crash vineyardd.

Commits

• [a67d9f19]: Add the interface to create blobs that backed by files on Disk. (#966) (Tao He)
• [3d716997]: Validate the etcd endpoint before feed into the etcd client. (#963) (Tao He)
• [cb3339f6]: Fixes a backwards compatibility issue in payload message. (#960) (Tao He)
• [4428f7a5]: Use the faster memcpy implementation from the ClickHouse project (#959) (Tao He)
• [ebb71386]: Add APIs for the access of offset array in the fragment (#958) (husimplicity)
• [fde982c8]: Replace arrow buffer with chunk buffer (#947) (liusitan)
• [1cece091]: Fixes a incorrect of shared_from_this in CRTP (#955) (Tao He)
• [f342b3a1]: Implement GPUBulkStore and GPUBulkAllocator (#876) (CSWater)
• [d77c3e26]: Fixes the compatibility issue with Ubuntu 18.04. (#950) (Tao He)

vineyard v0.7.2

Vineyard v0.7.2 introduces many new features

• Introduce a FetchAndGet api on client to access remote objects
• Introduce a command line flag --allocator which can be dlmalloc (by default) or mimalloc to select an allocator implementation
• Vineyard can be deployed as a systemd service (see also https://v6d.io/notes/deploy-locally.html#vineyard-a-systemd-service)

Many bugs has been fixed along with this release as well.

Commits

• [ccfb3cf3]: Installing vineyard as a systemd service (#949) (Tao He)
• [61f37417]: Leave larget space for mimalloc. (#946) (Tao He)
• [8c64a7ea]: Avoid including the unsealed blobs and the dummy blob into the result of listobjects. (#945) (Tao He)
• [cfe36d05]: Etcd launcher: use a temporary directory to data for each launching. (#942) (Tao He)
• [1789508f]: Ignore invalid metadata entries in client.ListData(). (#941) (Tao He)
• [ca825c5b]: Upgrage crdVersion to v1 and add multigroup (#939) (Ye Cao)
• [58a9c528]: Improve the ConstructFragmentGroup to accept projected fragments. (#934) (Tao He)
• [2f2c7f76]: vineyard fuse: add serilaization and deserialization factory (#889) (liusitan)
• [8e5643a2]: MarkAsCode: exclude "unsealed" blobs. (#933) (Tao He)
• [3301d5eb]: Add e2e test for scheduler-plugin. (#931) (Ye Cao)
• [a5b7475c]: Enable switch allocator using command line flags. (#930) (Tao He)
• [86d030a0]: Fixes the incorrect link in airflow provider's readme. (Tao He)
• [f11ef56f]: Refactor the migration APIs on vineyard client. (#927) (Tao He)
• [0a7ad1bd]: Cast boolean array to int array. (#928) (Tao He)
• [d0d64875]: Add docker images for vineyard installed and for vineyard developing (#929) (Tao He)
• [7e3a788c]: Unify the implementation of "mmap/munmap" in different allocators to avoid duplication. (#925) (Tao He)
• [66652cc1]: Address a missing import error in Python 3.8. (#924) (Tao He)
• [ecd09eb5]: Upgrade PreFilter to date, add sync_crds in helm charts, and remove unused ubuntu based dockerfile. (#922) (Ye Cao)
• [03e55b90]: "-D_GLIBCXX_USE_CXX11_ABI": allow override from command line. (Tao He)
• [8c8a0c65]: Disable usage of __rdtsc under certain circumstances. (#923) (Tao He)
• [b5749726]: Rename the flat_hash_map.hpp to avoid conflicts with other libraries. (#921) (Tao He)
• [cdd648f5]: Remove the jemalloc implementation and use mimalloc as the client side allocator. (#918) (Tao He)
• [3e88b434]: Infer the job replicas from the pods's ownership references (#917) (Tao He)
• [645dc77f]: Remove the unused vineyardjob CRD and refactor the scheduler plugin codebase (#915) (Tao He)
• [df8ea1c7]: Catch the json exception, especially inserting using json_pointer (#914) (Tao He)
• [4c276111]: Fixes the compatibility with latest nlohmann-json. (#909) (Tao He)
• [67883c14]: Upgrade k8s related components to v0.24.0 (#912) (Ye Cao)
• [af399eea]: Remove the API and implementation about "deep copy". (#910) (Tao He)

vineyard v0.7.1

Vineyard v0.7.1 is a bugfix release that improves the arrow-fragment-loader and related test cases a bit.

Commits

• [453778b5]: Improve the loaded_arrow_fragment_test for better argument parsing and output creation. (#903) (Tao He)
• [f0a4b9a0]: Add @dashanji to the maintainer list. (Tao He)
• [8a0b029d]: Update the copyright timeline (#901) (Ye Cao)
• [10ebb698]: Introduce golang lint (#900) (Ye Cao)
• [e0914954]: Fixes the typename parsing under gcc-7 with cxx11 ABI (on Ubuntu 18.04) (#898) (Tao He)
• [e2616d96]: Fixes a argument parsing bug in the vineyard-graph-loader. (#896) (Tao He)
• [22ce29a6]: Add benchmark test and result for mimalloc (#893) (Ye Cao)
• [54768dc3]: Keeps compatible with apache-arrow 9.0.0. (#895) (Tao He)
• [1ada72c6]: Use macos-11 runner on Github Actions. (Tao He)

vineyard v0.7.0

Commits

• [57a0d1d]: Add mimalloc for shared memory (#892) (Ye Cao)
• [1bfae06]: [CI] Don't panic if we just run out of memory. (#891) (Ke Meng)
• [bb5fd8b]: Return nullptr instead of throw exceptions in getobject. (#888) (Tao He)
• [6c1e52d]: Add a graph loader utility. (#887) (Tao He)

vineyard v0.6.2

Vineyard v0.6.2 is a bugfix release and bring prilimiary support for transparent spilling. The spill can be enabled with the vineyardd -spill_path command line argument.

Commits

• [25c064c]: Use more meaningful message when failed to create the IPC socket (#886) (Tao He)
• [a50b700]: Resolves the data race condition in the destruction of SocketServer. (#885) (Tao He)
• [a9ff904]: Improve the code quality according to the Python and C++ linters. (#882) (Tao He)
• [cad1c41]: fix a small error in the doc (#880) (dashanji)
• [a22c695]: Address some compilation warnings about warn-not-used. (#879) (Tao He)
• [0e2ef2b]: Cleanup the alloc_test, add a makefile for building and testing (#878) (Tao He)
• [af22e6f]: Support spill mechanism for v6d server (#815) (Yitao Wang)
• [7295b8e]: Add doc about etcd (#875) (dashanji)
• [36f62b0]: Add back the vineyard_{malloc,free,calloc,realloc} implementation. (#874) (Tao He)
• [021f5e7]: Split fragment into several compilation unit and conform the C++11 standard. (#872) (Tao He)
• [2ebd304]: Make sure the installed vineyard can be found using find_package() from cmake. (#871) (Tao He)
• [0377128]: Add a shared_ptr reference of bulkstore in the SocketConnection (#869) (Tao He)
• [0cb5726]: Allow customized image when deploying to Kubernetes from Python (#867) (Tao He)
• [c92885e]: Add header row for local_io_adaptor (#865) (wuyueandrew)
• [2a82aa7]: Leave spaces for jemalloc's arena metadata and internal padding (#866) (Tao He)
• [494d804]: Enhance the fuse integration. (#861) (Tao He)
• [c8a396c]: Fixes the wrong target class and missing arguments in the RPCClient. (#857) (Tao He)
• [32885f9]: Make sure the readme.rst conform the requirements of pypi (#855) (Tao He)
• [a8757ec]: Fixes the compatibility build. (Tao He)

vineyard v0.6.1

Vineyard v0.6.1 enables the following important experimental features:

• Limited mutatbe support for blobs, see also #843
• Refined documentation and deployment guidance on Kubernetes, see also: https://v6d.io/notes/deploy-kubernetes.html
• Integration with the hosseinmoein/DataFrame

Commits

• [c3000d3]: Add some debugging logs for reading tables/recordbatches from streams (#854) (Tao He)
• [6cd26e8]: Revisit the python API and documentation for Kubernetes deployment (#853) (Tao He)
• [5a30488]: Integration with hosseinmoein dataframe (#848) (vegetableysm)
• [84892fa]: Add documents for data accessing, streams and newly added APIs on clients (#850) (Tao He)
• [52b59af]: On M1 mac the error is a RuntimeError. (#847) (Tao He)
• [a48d007]: Enable mutable blob support. (#844) (Tao He)
• [312eda5]: Enable blob creation and access from RPC client. (#841) (Tao He)
• [a34bebf]: Revisit the implementation of is_shared_memory testing (#840) (Tao He)
• [bfb9afb]: make all GenerateBlobID use rand generator (#839) (Yitao Wang)
• [7133441]: Fix the bug that macosx thread explicitly marked deleted (#831) (Siyuan Zhang)
• [8ee97a1]: Force light theme, even the browers works under dark mode. (#832) (Tao He)
• [58d7f8c]: Refine the TOC of README.rst, and add committers to our maintainers.md. (#829) (Tao He)
• [cc6d2a0]: Add more sections for the concepts in vineyard and split the dive-in into several pages. (#827) (Tao He)
• [050b1fd]: Use SO_NOSIGPIPE for MacOS (#823) (Tao He)
• [4789f46]: Hotfix: Upgrade the numpy version to avoid compatiblity issue with pandas. (#819) (Tao He)
• [fc9611b]: Ensure the logo is centerized. (#817) (Tao He)
• [340cf25]: Update the vineyard documentation using the furo theme. (#816) (Tao He)

vineyard v0.6.0

Vineyard v0.6.0 enhances the support and backwards compatibility for airflow.

Commits

• [5156574]: Fixes the backwards compatibility issue of airflow (#814) (Tao He)
• [8e02498]: Makes libunwind as an optional dependency. (#809) (Tao He)
• [a37ed14]: Fixes the compatibility issue on MacOS. (#807) (Tao He)

vineyard v0.5.4

Vineyard v0.5.4 address several compliation issues about graph data structures.

Commits

• [7b5b144]: Fixes compilation failure of ArrowTensorT<string_view> with template specialization (#806) (DongZe Li)
• [d229431]: Fixes a typo in tutorials (#804) (bobhan1)
• [c5db621]: Enable optional wyhash for hashmap. (#800) (Tao He)
• [0da6a95]: Fixes the link failure under C++14. (#801) (Tao He)
• [924f694]: Support lru in ColdObjectTracker (#794) (Yitao Wang)

vineyard v0.5.3

Vineyard v0.5.3 handles several compatibility issue carefully to make vineyard work seamlessly on variaous platforms, e.g., CentOS and ArchLinux.

Commits

• [e45c987]: Add the contrib directory to vineyard include dirs. (#799) (Tao He)
• [c028225]: Erase the misleading tbb warning raised from intel-tbb directory. (#798) (Tao He)
• [0c16d2b]: Search libvineyard_internal_registry.so from current directory. (#797) (Tao He)
• [08dc6a1]: Handle LD_LIBRARY_PATH properly in runner.py for inplace building. (#796) (Tao He)

vineyard v0.5.2

Vineyard v0.5.2 has the following two key enhancement:

• Improve the serialization/deserialization to bump up the performance
• Add a new API to create placeholder on certain vineyard instance.