milvus-2.3.11

v2.3.11

Release date: Mar 8, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.11	2.3.6	2.3.4	2.3.5	2.3.5

Milvus v2.3.11 brings improvements and bug fixes aimed at enhancing performance, security, and stability. The improvements include optimized data loading, enhanced security with TLS support for Kafka connections, memory optimization, and more. Additionally, bug fixes address issues such as search/query failures, incorrect data types in outputs, and disk estimation errors. We encourage you to update to this latest version to take advantage of these enhancements and fixes.

Improvements

• Optimized JSON loading by reducing 1x memory copy (#30864)
• Implemented TLS support for Kafka connections (#30466 #30925)
• Optimized monitoring: Remove time tick delay metrics when nodes go offline (#30879)
• Optimized memory usage and loading speed for variable length data (#30900)
• Implemented support for varchar autoID in bulk insert operations (#30913)
• Added support for rate limiting for flush operations at the collection level (#29568)
• Improved compatibility for the watch DM channel request (#30954)
• Enhanced stability when reading S3 objects (#30976)
• Reduced contention for locks in the DescribeIndex operation (#30975)
• Optimized automatic balancing in QueryCoord (#30725)
• Enabled Milvus containers to run as a non-root user (#30937)
• Enhanced lock granularity in DataCoord meta (#30986)
• Enhanced memory estimation during loading by adding a load memory factor (#30999)
• Reduced memory consumption for DataNode when dealing with multiple collections (#30991)
• Optimized the connection manager in the proxy to prevent out-of-memory errors (#31009)

Bug Fixes

• Fixed search/query failed caused by passing an incorrect context and missing error handling (#30818)
• Fixed incorrect data types in the output when using the RESTful query interface (#30738)
• Fixed incorrect disk estimation when loading disk index (or index with mmap enabled) (#30948)
• Fixed the issue where redundant segments with older versions may never be released (#30953)
• Fixed the issue where incorrect disk limits may be read (#30966)
• Fixed a series of issues with the RESTful interface, such as incorrect int64 precision and failed insertion (#30873)
• Fixed panic during rolling upgrade caused by compatibility issues (#30656)
• Fixed an issue where flush and compaction were stalled due to uncleared compaction tasks in the Datanode (#30972)
• Fixed frequent lock contention in the GetCompactionTo operation, which caused channel checkpoints lag (#30965)
• Fixed the issue where Datanode failed to update channel checkpoints with multiple collections (#31024 #31082)
• Fixed insertion failures caused by frequent contention for locks (#31026)
• Fixed proxy panic caused by missing error handling (#31086)

milvus-2.3.10

v2.3.10

Release date: Feb 23, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.10	2.3.6	2.3.4	2.3.5	2.3.5

Milvus v2.3.10 is a critical patch release that follows Milvus v2.3.9, featuring several essential bug fixes. A significant fix in this release addresses the issue of missing data in hybrid search results when utilizing partition keys (issue #30607).

Users who have encountered hybrid search issues in release v2.3.5 through v2.3.9, especially those using partition key features, are highly recommended to upgrade to version v2.3.10 promptly.

Critical Bug Fixes

• Missing data in hybrid search results when utilizing partition keys (issue #30607).
• Prevented flush blockages by skipping the filling of segmentID in indexBuildCh (#30749).

Bug Fixes

• Prevented accidental deletion of original data by avoiding the use of absolute paths in ChunkCache (#30679).
• Enforced the use of virtual host for Tencent Cloud object storage (#30685).
• Updated disk usage metrics following segment release (#30707).
• Released loaded growing segments if WatchDmlChannel fails (#30745).
• Resolved a panic caused by using a nil interface in the gRPC client (#30755).

milvus-2.3.9

v2.3.9

Release date: Feb 19, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.9	2.3.6	2.3.4	2.3.5	2.3.5

Milvus 2.3.9 is a critical patch release succeeding Milvus 2.3.8, incorporating a number of vital bug fixes. Key among these is the resolution of the DiskANN construction failure issue. We strongly advise users experiencing disk index construction problems in version 2.3.8 to upgrade to version 2.3.9 without delay.

Critial Bug Fixes

• Fixed DiskANN construction failure in Milvus version 2.3.8 (#30640).
• Addressed loading failures due to inability to replace the primary key index (#30578).
• Added collectionName to the response of the ListAliases API (#30533).

milvus-2.3.8

v2.3.8

Release date: Feb 7, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.8	2.3.6	2.3.4	2.3.5	2.3.5

Milvus v2.3.8 is a minor patch release following Milvus v2.3.7. This release includes several enhancements and bug fixes. These improvements are designed to enhance system stability and observability. One of the key bug fixes is to prevent BulkInsert from getting stuck after a node restart.

Improvements

• Improved error messaging for dimension mismatch in search vectors (#30316)
• Integrated Milvus build process details, including commit information and dependency identifiers, into monitoring metrics (#29666)
• Streamlined loading strategy for segment Binlog files to optimize performance (#30348)
• Expanded BulkInsert feature to accommodate auto-incrementing primary keys for VarChar types (#30448)
• Enhanced memory estimation algorithm during data loading to prevent out-of-memory (OOM) errors (#30475)
• Eliminated extraneous log messages for cleaner logging (#30478)
• Updated to Knowhere version 2.2.4 for improved functionality (#30513)

Critical Bug Fixes

• Fixed panic error caused by watching multiple channels in the Datanodes (#30136)
• Corrected reading of index parameters from the configuration file (#30353)
• Ensured effectiveness of the db_name parameter for DescribeAlias and ListAliases operations (#30453)
• Resolved proxy startup hang-up due to improper port occupation handling (#30416)
• Refactored BulkInsert Flush process to prevent hang-ups after restart (#30439)

milvus-2.3.7

v2.3.7

Release date: Jan 29, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.7	2.3.6	2.3.4	2.3.5	2.3.5

Milvus v2.3.7 marks a minor yet impactful update, concentrating on boosting overall functionality and stability. In this release, we have refactored the business logic for a graceful stop to prevent any data loss, introduced support for array and JSON data types through RESTFul APIs, and amped up the speed of index loading. Alongside these enhancements, we've made several tweaks to optimize system performance and resource management. Additionally, this release addresses critical bug fixes as well as issues like memory leaks, load timeouts, and service unavailability, ensuring a more reliable and stable user experience.

Features

• Limit Collection Counts

  A Milvus instance allows up to 65,536 collections. However, too many collections may result in performance issues. Therefore, it is recommended to limit the number of collections created in a Milvus instance. Read more on Limit Collection Counts.

• Chunk Cache

  The chunk cache mechanism enables Milvus to pre-load data into cache memory on the local hard disk of the query nodes before it is needed. This mechanism significantly improves vector retrieval performance by reducing the time it takes to load data from disk to memory. Read more on Configure Chunk Cache

Improvements

• Transform specific magic numbers into configurable options (#30070).
• Remove heartbeat delay logic for ShardLeader to prevent misjudging its availability (#30085).
• When allocating channels, shuffle ShardLeader candidates to avoid load imbalance (#30089).
• Enhance RESTful support by adding functionality for arrays and JSON (#30077).
• Add a counter monitoring for rate-limited requests (#30132).
• Accelerate index loading through concurrent methods (#30018).
• Remove the step of DataNode subscribing to the message stream during the Import phase to avoid Import timeouts (#30133).
• Introduce association logic between privileges to simplify the authorization process (#30154).
• Implement unified restrictions on the number of Collections, Partitions, and Shards (#30017).
• Incorporate proactive pre-warming logic for ChunkCache to mitigate the issue of high latency when retrieving raw vectors during cold start queries (#30289)
• Optimize the load balancing algorithm by assigning weight to growing segments (#30293)
• Remove unnecessary business logic for conversions between partition names and IDs to reduce latency in the data retrieve stage when processing search requests (#30255)

Critical Bug Fixes

• Fixed a memory leak caused by incorrect usage of OpenTelemetry in the Segcore (#30068).
• Addressed the issue of slow disk index loading by dynamically patching the index parameters (#30116).
• Resolved the problem of changes made through the "alter collection" command not being persisted (#30156).
• Fixed the issue where read request rate limiting ultimately leads to the unavailability of the read service (#30196).
• Resolve the deadlock issue when getting the configuration (#30319)
• Fix incorrect usage of the Goroutine pool on CGO calls (#30275)
• Add a timeout mechanism to the graceful stop process to prevent potential cases of getting stuck(#30320)

milvus-2.3.5

v2.3.5

Release date: Jan 17, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.5	2.3.5	2.3.4	2.3.5	2.3.5

Milvus v2.3.5 marks a significant update focused on improving overall functionality and stability. In this release, we've made enhancements to Role-Based Access Control (RBAC) and Alias, prioritizing user-friendliness. Additionally, various improvements have been implemented to optimize system performance and resource management. A notable feature in v2.3.5 is the reintroduction of the MVCC (Multi-Version Concurrency Control) mechanism, crucial for efficiently managing concurrent and phased queries. This release also addresses critical bug fixes, preventing issues related to data deletion loss, abnormal system panics, and deadlocks, ensuring a more reliable and stable user experience.

Features

• Role-Based Access Control (RBAC)
  • Authorize users to query grant information for their roles. (#29747)
  • Feature: Add RBAC functionality to alias. (#29885)

Improvements:

• Restore MVCC functionality (#29749)
• Add concurrency for DataCoord segment GC to increase garbage collection speed (#29557)
• Read Azure files without ReadAll to control memory usage (#29604)
• Support reading hardware metrics for cgroupv2 (#29847)
• Save lite WatchInfo into etcd in DataNode (#29751)
• Support access log printing cluster prefix (#29646)
• Rewrite generation segment plan based on assigning segments to make it more understandable (#29574)
• Performance:
  • Enhancement: Use GPU pool for GPU tasks (#29678)
  • Cache collection schema attributes to reduce proxy CPU usage (#29668)
  • Pre-allocate result FieldData space to reduce growing slices (#29726)

Critical Bug Fixes:

• Fix the delete message disorder issue causing data loss (#29917)
• Throw an exception when the upload file fails for DiskIndex to avoid core dump (#29628)
• Fix dynamic update of rate limit config with incorrect value (#29902)
• Ensure compact operation on DataCoord meta performs as a transaction (#29776)
• Fix panic caused by type assert LocalSegment on Segment (#29018)
• Drop segment meta info with a prefix to avoid etcd txn out of limit (#29857)
• Remove unnecessary lock-in config manager (#29855)
• Rectify memory leaks when reading data from Azure. (#30006)
• Resolve the issue of mistakenly duplicating dynamic fields when handling RESTful insert requests. (#30043)
• Rectify the deadlock issue in the BlockAll operation within the flowgraph. (#29972)
• Resolve the issue of failing to parse lengthy and complex expressions. (#30021)

milvus-2.3.4

2.3.4

Release date: Jan 2, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.4	2.3.5	2.3.3	2.3.4	2.3.5

Milvus 2.3.4 brings significant enhancements, focusing on availability and usability. The update introduces access logs for better monitoring and integrates Parquet for efficient bulk imports. A key feature is the binlog index on growing segments for faster searches. Major improvements include support for up to 10,000 collections/partitions, reduced memory usage, clearer error messages, quicker loading, and better query shard balance. It addresses critical issues like resource leakage, load/release failures, and concurrency challenges. However, it discontinues regular expression searches in partitions to save resources, with an option to re-enable this feature in the configuration.

Features

• Access Logs:

  • Milvus now supports access logs for monitoring external interfaces. These logs record method names, user requests, response times, and error codes.
  • Note: Currently, this feature supports only gRPC; RESTful requests are not included.

• Parquet File Import:

  • This update introduces support for Parquet file imports, enhancing performance and memory efficiency. It also broadens data type support, including arrays and JSON.
  • This feature supersedes the previous limitation of JSON and NumPy formats.

• Binlog Index on Growing Segments:

  • Milvus now employs a binlog index on growing segments to enhance search efficiency, allowing for advanced indices like IVF or Fast Scann.
  • This improvement can increase search speeds in growing segments by up to tenfold.

Improvements

• Expanded Collection/Partition Support:

  • Milvus now supports up to 10,000 collections/partitions in a cluster, benefiting multi-tenant environments.
  • The improvement comes from timetick mechanism refinement, goroutine management, and memory usage improvement.
  • Note: Exceeding the recommended limit may affect failure recovery and resource usage. Recommended limit is 10,000 (Collection * Shard * Partition).

• Reduced Memory Usage:

  • Enhancements have been made to improve memory efficiency during various operations, including data retrieval and variable length data handling.

• Refined Error Messaging:

  • Error messages have been split into summaries and details for clearer understanding.

• Accelerated Loading Speed:

  • Various optimizations have been implemented to increase loading speeds, particularly in scenarios with frequent flushes and deletions.

• Improved Query Shard Balance:

  • Implemented balance channel in querycoord and other improvements for efficient shard management.

• Other Enhancements:

  • Includes security improvements, MMap support for index loading, partition-level privileges, and more.

Critical Bug Fixes

• Resource Leakage Fixes:

  • Addressed critical memory leaks in Pulsar producer/consumer and improved garbage collection of meta snapshots.

• Load/Release Failure Fixes:

  • Resolved issues causing load/release operations to stall, especially in clusters with many segments.

• Concurrency Issues:

  • Fixed problems related to concurrent insertions, deletions, and queries.

• Other Critical Fixes:

  • Fixed an issue where upgrades from version 2.2 failed due to missing CollectionLoadInfo.
  • Fixed an issue where deletions might be lost because of errors in parsing compacted file logpaths (#29276).
  • Fixed an issue where flush and compaction processes could become stuck under heavy insert/delete traffic.
  • Fixed the inability to perform compact operations on the array type (#29505) (#29504).
  • Fixed an issue where collections with more than 128 partitions failed to be released (#28567).
  • Fixed an issue related to parsing expressions that include quotation marks (#28418).
  • Addressed a failure in Azure Blob Storage's ListObjects operation causing garbage collection failures (#27931) (#28894).
  • Fixed an issue with missing target database names in RenameCollection operations (#28911).
  • Fixed an issue where iterators lost data in cases of duplicated results (#29406) (#29446).
  • Corrected the bulk insert binlog process to consider timestamp order when processing delta data (#29176).
  • Fixed an issue to exclude insert data before a growing checkpoint (#29559).
  • Addressed a problem where frequent flushing caused rate limits in Minio (#28625).
  • Fixed an issue where creating growing segments could introduce an excessive number of threads (#29314).
  • Fixed an issue in retrieving binary vectors from chunk cache (#28866) (#28884).
  • Fixed an issue where checkpoints were incorrectly updated after dropping a collection (#29221).

Breaking Change

• Discontinued Regular Expression Search in Partitions:
  • To reduce resource consumption, regular expression searches in partitions have been discontinued. However, this feature can be re-enabled through configuration (see #29154 for details).

milvus-2.2.16

2.2.16

Release date: Nov 27, 2023

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.2.16	2.2.17	2.2.15	2.2.8	2.2.24

Milvus 2.2.16 represents a minor patch release following Milvus 2.2.15. This update primarily concentrates on bolstering system stability, enhancing fault recovery speed, and addressing various identified issues. Notably, the Knowhere version has been updated in this release, leading to quicker loading of DiskAnn indexes.

For an optimal experience, we highly recommend all users currently on the 2.2.0 series to upgrade to this version before considering a move to 2.3.

Bug Fixes

• Corrected the docker-compose etcd health check command (27980).
• Completed the cleanup of remaining meta information after dropping a Collection (28500).
• Rectified the issue causing panic during the execution of stop logic in query coordination (28543).
• Resolved the problem of the cmux server failing to gracefully shut down (28384).
• Eliminated the reference counting logic related to the query shard service to prevent potential leaks (28547).
• Removed the logic of polling collection information from RootCoord during the restart process of QueryCoord to prevent startup failures (28607).
• Fixed parsing errors in expressions containing mixed single and double quotations (28417).
• Addressed DataNode panic during flushing delete buffer (28710).

Enhancements

• Updated Knowhere to version 1.3.20 to accelerate the loading process (28658).
• Made etcdkv request timeout configurable (28664).
• Increased the timeout duration for QueryCoord to probe the query nodes via gRPC to 2 seconds (28647).
• Bumped milvus-proto/go-api to version 2.2.16 (28708).

milvus-2.2.15

2.2.15

Release date: Nov 10, 2023

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.2.15	2.2.17	2.2.15	v2.2.8	2.2.24

Milvus 2.2.15, a bugfix version of the Milvus 2.2.x series, has introduced significant improvements and bug fixes. This version enhanced the bulkinsert functionality to support partitionkey and the new JSON list format. Additionally, 2.2.15 has substantially improved the rolling upgrade process to 2.3.3 and resolved many critical issues. We strongly recommend all 2.2.15 users upgrade to this version before moving to 2.3.

Incompatible Update

• Removed MySQL metastore support (#26634).

Features

• Enabled bulkinsert of binlog data with partitionkey (#27336).
• Added support for bulkinsert with pure list JSON (#28127).

Improvements

• Added -g flag for compiling with debug information (#26698).
• Implemented a workaround to fix ChannelManager holding mutex for too long (#26870, #26874).
• Reduced the number of goroutines resulting from GetIndexInfos (#27547).
• Eliminated the recollection of segment stats during datacoord startup (#27562).
• Removed flush from the DDL queue (#27691).
• Decreased the write lock scope in channel manager (#27824).
• Reduced the number of parallel tasks for compaction (#27900).
• Refined RPC call in unwatch drop channel (#27884).
• Enhanced bulkinsert to read varchar in batches (#26198).
• Optimized Milvus rolling upgrade process, including:
  • Refined standalone components' stop order (#26742, #26778).
  • Improved RPC client retry mechanism (#26797).
  • Handled errors from new RootCoord for DescribeCollection (#27029).
  • Added a stop hook for session cleanup (#27565).
  • Accelerated shard leader cache update frequency (#27641).
  • Disabled retryable error logic in search/query operations (#27661).
  • Supported signal reception from parent process (#27755).
  • Checked data sync service number during graceful stop (#27789).
  • Fixed query shard service leak (#27848).
  • Refined Proxy stop process (#27910).
  • Fixed deletion of session key with prefix (#28261).
  • Addressed unretryable errors (#27955).
  • Refined stop order for components (#28017).
  • Added timeout for graceful stop (#27326, #28226).
  • Implemented fast fail when querynode is not ready (#28204).

Bug Fixes

• Resolved CollectionNotFound error during describe rg (#26569).
• Fixed issue where timeout tasks never released the queue (#26594).
• Refined signal handler for the entire Milvus role lifetime (#26642, #26702).
• Addressed panic caused by non-nil component pointer to component interface (#27079).
• Enhanced garbage collector to fetch meta after listing from storage (#27205).
• Fixed Kafka consumer connection leak (#27223).
• Reduced RPC size for GetRecoveryInfoV2 (#27484).
• Resolved concurrent parsing expression issues with strings (#26721, #27539).
• Fixed query shard inUse leak (#27765).
• Corrected rootPath issue when querynode cleaned local directory (#28314).
• Ensured compatibility with sync target version (#28290).
• Fixed release of query shard when releasing growing segment (#28040).
• Addressed slow response in flushManager.isFull (#28141, #28149).
• Implemented check for length before comparing strings (#28111).
• Resolved panic during close delete flow graph (#28202).
• Fixed bulkinsert bug where segments were compacted after import (#28200).
• Solved data node panic during save binlog path (#28243).
• Updated collection target after observer start (#27962).

milvus-2.3.3

2.3.3

Release date: Nov 10, 2023

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.3	2.3.4	2.3.3	2.3.3	2.3.3

Features

Supported pure list JSON in bulk insert (#28126)

Improvements

• Constructed a plan directly when searching with vector output (#27963)
• Removed binlog/delta log from getRecoveryInfoV2 (#27895) (#28090)
• Refined code for fixed-length types array (#28109)
• Improved rolling upgrade unserviceable time
  • Refined stop order (#28016) (#28089)
  • Set qcv2 index task priority to Low (#28117) (#28134)
  • Removed retry in getShards (#28011) (#28091)
  • Fixed load index for stopping node (#28047) (#28137)
  • Fixed retry on offline node (#28079) (#28139)
  • Fixed QueryNode panic while upgrading (#28034) (#28114)
  • Fixed coordinator fast restart by deleting old session (#28205)
  • Fixed check grpc error logic (#28182) (#28218)
  • Delayed the cancellation of ctx when stopping the node (#28249)
  • Disabled auto balance when an old node exists (#28191) (#28224)
  • Fixed auto balance block channel reassign after datanode restart (#28276)
  • Fixed retry when proxy stopped (#28263)
• Reduced useless ObjectExists in AzureBlobManager (#28157)
• Got vector concurrently (#28119)
• Forced set Aliyun use_virtual_host to true for all (#28237)
• Fixed delete session key with prefix causing multiple QueryNode crashes (#28267)

Bug Fixes

• Fixed script stop unable to find Milvus process (#27958)
• Fixed timestamp reordering issue with delete records (#27941) (#28113)
• Fixed prefix query with longer subarray potentially causing a crash (#28112)
• Limited max thread num for pool (#28018) (#28115)
• Fixed sync distribution with the wrong version (#28130) (#28170)
• Added a custom HTTP header: Accept-Type-Allow-Int64 for JS client (#28125)
• Fixed bug for constructing ArrayView with fixed-length type (#28186)
• Fixed bug for setting index state when IndexNode connecting failed (#28221)
• Fixed bulk insert bug that segments are compacted after import (#28227)
• Fixed the target updated before version updated to cause data missing (#28257)
• Handled exceptions while loading (#28306)