Skip to content

Release v0.29.0

Latest
Latest
Compare
Choose a tag to compare
@lidongze0629 lidongze0629 released this 29 Sep 09:50
· 39 commits to main since this release
v0.29.0
c3b589d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

We are delighted to bring a number of improvements to GraphScope, alongside the GraphScope 0.29.0 release. This release contains many important features and improvements to GraphScope FLEX、gsctl command-line utility、Graph Interactive Engine (GIE) and Groot persistent storage.

We highlight the following improvements included in this release:

1. gsctl command-line utility
gsctl is a command-line utility for GraphScope. It provides a set of functionalities to make it easy to use GraphScope. These functionalities include building images and packages, managing sessions and resources, and more.

Install/Update gsctl

$ pip3 install gsctl
# or force reinstall gsctl by:
$ pip3 install gsctl --force-reinstall -U

The gsctl command-line utility supports two modes of operation: utility scripts and client/server mode. You can switch between these modes using the gsctl connect and gsctl close commands.

Utility Scripts
Default, the gsctl provide helper functions and utilities that can be run using gsctl alone. gsctl acts as the command-line entrypoint for GraphScope. Some examples of utility scripts are:

  • gsctl install-deps, install dependencies for building GraphScope.
  • gsctl connect, connect to the launched coordinator with configuration file ~/.gsctl.
  • gsctl close, Close the connection from the coordinator.
  • gsctl flexbuild, Build docker image for Interactive, Insight product.
  • gsctl version, Print the client version information.
  • gsctl instance, Deploy, destroy a GraphScope Flex instance.

Client/Server Mode
To switch to the client/server mode, use the gsctl connect command. By default, this command connects gsctl to a launched coordinator using the configuration file located at ${HOME}/.gsctl; If --coordinator-endpoint parameter is specified, it will treat it as current context and override the configuration file.

Once connected, you can use gsctl to communicate with the coordinator which serves the specific Flex product behind it.

$ gsctl use GRAPH modern_graph
Using GRAPH modern_graph

2. 🎉 New Product: Graphscope Portal

We are delighted to announce the release of GraphScope Portal, a user-friendly web interface designed to simplify graph data management with GraphScope. It offers one-stop access to data modeling, importing, querying, and fully supports the Interactive computing engine with the GraphScope Flex architecture.For detailed usage, please refer to: https://github.com/GraphScope/portal

gs-portal

3. FLEX Interactive

  • Interactive now supports executing ad-hoc Cypher queries directly, without the need to compile them into stored procedures. Users can install Interactive, submit Cypher queries, and immediately experience improved query performance.
  • Offers the ability to create stored procedures using C++.
  • Enhance the ability of gsctl on Interactive. Including support customizing the configuration of service, registering cpp stored procedures via gsctl.
  • Adapt to the unified error code of GraphScope.
  • Improve the robustness and stability.

4. Enhancements for GIE

  • New Functionalities:
    • Support PathExpand with the TRAIL option. Currently, we support PATH_OPT values: ARBITRARY (allowing the duplication of vertices or edges), SIMPLE (no duplicated nodes), and TRAIL (no duplicated edges).
    • Support elementMap() for path in Gremlin, to project id, label and properties of each element in path.
    • Support Optional Match in Cypher to search for the pattern described in it, while using nulls for missing parts of the pattern.
    • Support UNWIND in Cypher to unfold elements from a collection type.

More details can be referred in Cypher Support Doc and Gremlin Support Doc.

  • Improve robustness and stability through extensive tests of the GOpt-based compilation stack.

  • User-Friendly Improvements:

    • Introducing a new benchmarking tool for comparing GIE with other systems, which supports multiple query languages such as Cypher and Gremlin, and handles versatile workloads including LDBC IC and BI, LSQB, and JOB. For more information, please refer to the Benchmark Tool Doc.
    • Adaptation to the unified error code of GraphScope.

5. Bug Fixes

  • Resolve the issue with the Java app's output.
  • Resolve some memory leak problem of GAE Java SDK.
  • Fixed a memory leak when reopening secondary groot instance.
  • Fixed some bugs in type inference in GOpt
  • Fixed some bugs in schema/statistics fetching in GOpt
  • Fixed some bugs in caridinality estimation in GOpt
  • Fixed result parsing for map structure in GIE
  • Fixed precedence bug in Predicates expression evaluation in GIE

Docker Images

# Flex Interactive image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/interactive:0.29.0

# Coordinator Image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/coordinator:0.29.0

# Graph Analytical Engine(GAE) Image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/analytical:0.29.0

# Frontend component of Graph Interactive Engine(GIE) image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/interactive-frontend:0.29.0
# Executor component of Graph Interactive Engine(GIE) image 
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/interactive-executor:0.29.0

# Graph Learning Engine(GLE) Image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/learning:0.29.0

# GraphScope persistent storage, user can only perform GIE query on it.
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-store:0.29.0

# Develop Image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-dev:latest

What's Changed

  • refactor(interactive): Add dev_and_test tutorial for Interactive by @zhanglei1949 in #3997
  • fix(interactive): Fix metadata getting for loading config by @zhanglei1949 in #3915
  • fix: make Groot compatible with jdk1.8 by @lidongze0629 in #4004
  • fix(interactive): Give extra params such as snapshot id in the physical plan by @BingqingLyu in #3999
  • fix(interactive): Support long_text as primary key by @zhanglei1949 in #4011
  • fix(interactive): Support elementMap() for a path entry in GOpt by @BingqingLyu in #3992
  • fix(interactive): Fix bug of loading edges with long_text property by @zhanglei1949 in #4018
  • fix(interactive): Fixing error messages receiving in Python SDK. by @zhanglei1949 in #4023
  • fix(interactive): Add graph_id/proc_id type check for python sdk by @zhanglei1949 in #4024
  • fix(interactive): Enhance input file path parsing in loadingConfig by @zhanglei1949 in #4026
  • refactor(interactive): Add method to get edge_num and some code refactor by @zhanglei1949 in #4029
  • fix(interactive): Fix Result Parsing Mismatch Errors of Map Structure by @shirly121 in #4006
  • fix(interactive): Add meta data information of IMDB dataset by @BingqingLyu in #4015
  • fix(interactive): Support parsing quoting_char when reading header row from csv by @zhanglei1949 in #4031
  • refactor(interactive): Refine Interactive SDK documentation by @zhanglei1949 in #3978
  • refactor(interactive): Refine the example cypher stored procedures by @zhanglei1949 in #4044
  • fix(interactive): refine the logs in ExpStore dataloading by @BingqingLyu in #4042
  • fix(interactive): fix bug in hashcode() for PatternOrder by @BingqingLyu in #4013
  • fix: get stored procedure failed with long_text type by @lidongze0629 in #4045
  • docs: Update README.md by fixing typos. by @Jiaohm in #4007
  • fix(analytical): add check direction for triangles by @siyuan0322 in #4049
  • refactor(interactive): Remove some unnecessary logging by @zhanglei1949 in #4055
  • chore: bump setuptools from 65.7.0 to 70.0.0 in /coordinator by @dependabot in #4046
  • fix: No module named 'graphscope.flex.rest.api' in gsctl by @lidongze0629 in #4061
  • fix(analytical): involving triangles by @siyuan0322 in #4067
  • refactor(interactive): Order procedures by default based on creation time. by @zhanglei1949 in #4059
  • refactor(interactive): Fix some CI problem and code support procedure-query-only mode by @zhanglei1949 in #4068
  • fix(interactive): Improve the codegen error propagation from server to client by @zhanglei1949 in #4062
  • fix(interactive): fix get empty store state error by @siyuan0322 in #4071
  • refactor(interactive): No need to restart compiler when restarting interactive by @zhanglei1949 in #4076
  • fix(interactive): Interactive CI hot fix by @zhanglei1949 in #4079
  • fix(interactive): Standardized log for sls metric by @bufapiqi in #4073
  • fix(interactive): Fix the ignoring of protoc-generated files for interactive python sdk by @zhanglei1949 in #4053
  • fix(interactive): fix bulkloading data from csv files without header and add null_values option for arrow by @liulx20 in #4083
  • fix(interactive): fix ci by @liulx20 in #4084
  • refactor(interactive): Reorganize the http handler of interactive by @zhanglei1949 in #4037
  • fix(analytical): Resolve the issue of the Java app's results not being outputted by the context. by @zhanglei1949 in #4082
  • fix(interactive): Fix early stop in pegasus when unblock other output by @lnfjpt in #4086
  • fix(interactive): Fix the problem of actor scope cancellation and creation by @zhanglei1949 in #4089
  • fix(interactive): move some statistics files by @BingqingLyu in #4065
  • chore: bump setuptools from 65.7.0 to 70.0.0 in /python by @dependabot in #4066
  • fix(interactive): Fix multiple match problem when deserializing json string to GSDataType by @zhanglei1949 in #4091
  • chore: update GitHub runner version by @siyuan0322 in #4103
  • feat(interactive): Add Create/Update/Get APIs for vertex/edge in Interactive by @yqylh in #4025
  • feat(interactive): support adhoc query by @liulx20 in #4094
  • fix(interactive): add orderBy operator for graphAlgo test case by @liulx20 in #4030
  • refactor: Reorgnize error codes and unify the format of error message. by @siyuan0322 in #4100
  • fix(interactive): add some experimental queries in paper by @BingqingLyu in #4110
  • fix(interactive): Fix receiver state error after reconnect in a short time by @lnfjpt in #4097
  • docs: Update CITATION to latest GraphScope Flex paper. by @yecol in #4113
  • refactor(interactive): Replace CHECK with std::runtime_error in schema.cc by @zhanglei1949 in #4107
  • refactor(interactive): Add test to ensure the robustness of Interactive by @zhanglei1949 in #4078
  • feat: refactor the coordinator to support Groot under the unified schema architecture by @lidongze0629 in #4087
  • refactor(interactive): Refactor interactive image building by @zhanglei1949 in #4056
  • fix(interactive): Support multiple properties edge expand by @liulx20 in #4115
  • fix(interactive): fix bulking load with duplicate vertex id by @liulx20 in #4120
  • refactor(interactive): Reduce the size of codegen library by @zhanglei1949 in #4118
  • refactor(interactive): Remove some unnecessary check by @zhanglei1949 in #4116
  • fix(interactive): add error message for file operations by @liulx20 in #4126
  • fix(interactive): fix bugs in physical plan converter in GOpt by @BingqingLyu in #4109
  • fix(interactive): fix bug when concating simple path in GIE Runtime by @BingqingLyu in #4102
  • fix(interactive): Expose config for Glogue size and pattern decomposition size in Compiler in Charts by @BingqingLyu in #4081
  • fix(interactive): fix query cardinality of non-exist types or patterns by @BingqingLyu in #4080
  • fix(interactive): Accelerate replay speed. by @siyuan0322 in #4130
  • fix(interactive): Make Statistics Configurable by Interface in Compiler by @shirly121 in #4123
  • fix(interactive): Fix Bugs of Property Loss of Edge Type After Type Inference by @shirly121 in #4032
  • fix(interactive): Catch exceptions thrown by nlohmann json by @zhanglei1949 in #4127
  • fix: Fix CI failure when building manylinux image by @lidongze0629 in #4122
  • fix(interactive): fix sinking edge without properties. by @liulx20 in #4139
  • fix(interactive): Fix files not closing properly after the db closed. by @liulx20 in #4136
  • refactor: deprecate the legacy Jupyter extension by @yecol in #4140
  • ci: Add github codeql to scan code. by @yecol in #4135
  • chore(k8s): Add a tip about connecting with minikube by @Thespica in #4141
  • ci: Refine docs deployment and PR check CI by @yecol in #4148
  • ci: make codeql run less frequent by @yecol in #4149
  • chore: add affinity to graphscope-store charts by @siyuan0322 in #4155
  • fix(interactive): refine Insight server error by @BingqingLyu in #4152
  • refactor(interactive): Introduce GOpt Doc by @shirly121 in #4157
  • ci: Update pr-check.yml by @yecol in #4160
  • fix(interactive): add runtime test tool by @liulx20 in #4156
  • fix(interactive): fix some bug with GroupBy by @liulx20 in #4165
  • feat(interactive): Add the CMake argument OPTIMIZE_FOR_HOST to prevent the use of host-specific CPU instructions when building images. by @zhanglei1949 in #4159
  • feat(interactive): add storage based on csr by @lnfjpt in #4150
  • fix(interactive): fix some bug with Intersect by @liulx20 in #4168
  • fix(interactive): Add max retry for ODPS fragment loader by @zhanglei1949 in #4166
  • fix(interactive): fix zero property edge and schema check by @yqylh in #4131
  • refactor(interactive): Adapt to unified error code by @zhanglei1949 in #4128
  • fix(interactive): Fix Bugs in Dynamic Schema Fetching by @shirly121 in #4176
  • fix(interactive): prepare more graphs for groot in ci tests by @BingqingLyu in #4180
  • fix(interactive): vertex with string oid is not support by @liulx20 in #4171
  • fix(interactive): Shade Scala Package in Neo4j to Avoid Conflicts by @shirly121 in #4175
  • fix: Improve the robustness of code by @FLYLX in #4170
  • feat(interactive): Support PathExpand with PATH_OPT=TRAIL in GIE by @fangfcg in #4101
  • fix(interactive): Add support for CBO on Interactive and fix some codgen errors by @zhanglei1949 in #3660
  • Revert "feat(interactive): Support PathExpand with PATH_OPT=TRAIL in GIE" by @shirly121 in #4183
  • feat(interactive): Support PathExpand with PATH_OPT=TRAIL in GIE by @fangfcg in #4184
  • fix(analytical): Fix memory leak for FFI Objects by @zhanglei1949 in #4185
  • refactor(interactive): Pass worker_id as a parameter instead of lazy_static by @lnfjpt in #4060
  • refactor(interactive): Reduce Interactive image size by @zhanglei1949 in #4178
  • fix(python): fix protobuf version is less than 5 error in mini test by @siyuan0322 in #4186
  • docs(interactive): Add customized deployment documentation by @zhanglei1949 in #4179
  • fix(interactive): Unify Error Code in Compiler by @shirly121 in #4182
  • refactor(interactive): Add Encoder/Decoder for Python SDK by @zhanglei1949 in #4191
  • fix(interactive): Fix robust test by @zhanglei1949 in #4193
  • fix(interactive): Fix Bugs of Count Estimation When Feasible Plan Is Unavailable in Equivalent Set by @shirly121 in #4181
  • feat: Reorganize install-deps in gsctl by @lidongze0629 in #4154
  • refactor(interactive): Rename the configuration file for Interactive, and remove some unused configuration by @zhanglei1949 in #4202
  • fix(interactive): Refine Count Estimation of CBO to Achieve Optimal Plan When Integrating GOpt Into Neo4j by @shirly121 in #4198
  • chore: fix several compile errors with downstream repos. by @siyuan0322 in #4196
  • fix(interactive): Unified Query Cache for Gremlin and Cypher Queries by @shirly121 in #3890
  • feat(interactive): Support customizing service configuration via gsctl by @zhanglei1949 in #4205
  • refactor(interactive): Add error handling for adhoc queries by @zhanglei1949 in #4188
  • fix: upgrade boost version to v1.75.0 for leaf by @lidongze0629 in #4211
  • fix: remove openapi generated code by @lidongze0629 in #4214
  • fix(interactive): Fix Bugs of Type Inference in both() by @shirly121 in #4199
  • fix(interactive): Fixing tutorial examples for Interactive by @zhanglei1949 in #4212
  • docs(interactive): Refine the documentation of Interactive by @zhanglei1949 in #4209
  • refactor(interactive): Eliminate dummy installation in the Dockerfile. by @zhanglei1949 in #4218
  • refactor(interactive): Replacing nlohmann-json with rapidjson by @yqylh in #4194
  • refactor(interactive): Support Fetching Meta By Calling Procedure in Compiler by @shirly121 in #4215
  • fix(interactive): Fix description passing for procedure creation by @zhanglei1949 in #4223
  • fix(interactive): Fix bug in scan operator by @zhanglei1949 in #4221
  • feat: Support creating c++ stored procedure via gsctl by @lidongze0629 in #4222
  • fix(interactive): Fix typos by @co63oc in #4224
  • fix(interactive): Correct the number of edge count by @zhanglei1949 in #4229
  • refactor(interactive): Refactor the CMake configuration by @zhanglei1949 in #4217
  • docs: update coordinator and gsctl doc by @lidongze0629 in #4232
  • ci: integrate install dependencies script of Interactive into gsctl by @lidongze0629 in #4234
  • fix(interactive): Support more cases for var in expression by @zhanglei1949 in #4230
  • fix(interactive): Update flex/Dockerfile by @zhanglei1949 in #4236
  • fix(interactive): Fix Bugs of Pattern Not Found in Glogue by @shirly121 in #4235
  • docs(interactive): Add doc for creating cpp procedure with gsctl by @zhanglei1949 in #4231
  • feat(interactive): Introduce new version of graph_algo example graph by @zhanglei1949 in #4153
  • ci: fix ci problem during release gsctl by @lidongze0629 in #4238
  • feat(interactive): Support optional expand in GIE Runtime by @BingqingLyu in #4213
  • feat(interactive): Support builtin app in Interactive by @zhanglei1949 in #4242
  • fix(interactive): Reuse the graph schema preprocessing to create the default graph schema when the service starts by @zhanglei1949 in #4239
  • fix(interactive): fix Predicates to correctly process precedence in expression by @BingqingLyu in #4041
  • refactor(interactive): Support Unfold Operation in Cypher Queries by @shirly121 in #4241
  • fix(interactive): Fix memory leak when reopening rocksdb instance by @siyuan0322 in #4250
  • refactor(interactive): Support User Defined Functions in Compiler by @shirly121 in #4240
  • feat(interactive): Introduce a new benchmark tool for GIE by @BingqingLyu in #4245
  • fix(interactive): add cis for Insight with GOpt by @BingqingLyu in #4200
  • ci: improve codecov coverage by @lidongze0629 in #4251
  • chore: exlude simple and log4j jar from groot assembly by @siyuan0322 in #4254
  • chore: fix log configuration format by @siyuan0322 in #4255
  • chore: refine logs of groot by @siyuan0322 in #4258
  • feat(interactive): Introduce imdb as another example graph for Interactive by @zhanglei1949 in #4208
  • fix(interactive): Fix the behavior the service restarting from workspace by @zhanglei1949 in #4259
  • fix: codecov uploading problem by @lidongze0629 in #4257
  • ci: release interactive image nightly by @lidongze0629 in #4262
  • fix: hot fix for building interactive image by @lidongze0629 in #4264
  • fix(interactive): Some fix in the Benchmark Tool by @BingqingLyu in #4261
  • fix: install-deps fails on macOS(intel) platform by @lidongze0629 in #4266
  • refactor(interactive): Refine the documentation of Interactive SDK by @zhanglei1949 in #4268
  • fix(interactive): Fix Concurrent Bugs in Graph Optimizer by @shirly121 in #4269
  • fix(interactive): refine the logs when fetching statistics from groot by @BingqingLyu in #4267
  • fix: fails of building learning image by @lidongze0629 in #4273

New Contributors

Full Changelog: v0.28.0...v0.29.0

Contributors

yecol, co63oc, and 14 other contributors
Assets 2
Loading