milvus-2.4.13-hotfix

v2.4.13-hotfix

Release Date: October 17, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.13-hotfix	2.4.8	2.4.5	2.4.9

Milvus v2.4.13-hotfix addresses a critical issue specific to v2.4.13, where Milvus may fail to retrieve collection information after a restart if all MetaKV snapshots were garbage-collected (#36933). Users currently running v2.4.13 are advised to upgrade to v2.4.13-hotfix at the earliest opportunity to avoid potential disruptions.

Critical fixes

• Load original key if timestamp is MaxTimestamp (#36935)

milvus-2.4.13

v2.4.13

Release Date: October 12, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.13	2.4.8	2.4.5	2.4.9

Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection. This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery. Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency. We highly recommend upgrading to this release for better performance and stability.

Features

• Dynamic replica adjustment for loaded collections (#36417)
• Sparse vector MMAP in growing segment types (#36565)

Bug fixes

• Fixed a flush performance issue (#36741)
• Fixed a bug with JSON expressions in "[]" (#36722)
• Removed neighbors if compact target is unindexed (#36694)
• Improved performance for Rocksmq when channel is full (#36618)
• Fixed an issue where errors during unpinning were not deferred (#36665)
• Resolved a memory leak for imported segments in the segment manager (#36631)
• Skipped unnecessary health checks for query nodes in the proxy (#36553)
• Fixed an overflow issue with term expressions (#36534)
• Recorded node ID before assigning tasks to prevent task misallocation (#36493)
• Resolved data race issues in clustering compaction (#36499)
• Added a check for string array max length after type matching (#36497)
• Addressed race conditions in mix or standalone mode (#36459)
• Fixed segment imbalance after repeated load and release operations (#36543)
• Corrected a corner case where segments couldn't be moved from a stopping node (#36475)
• Updated segment info properly even if some segments were missing (#36729)
• Prevented etcd transactions from exceeding the max limit in snapshot KV (#36773)

Improvements

• Enhanced MMAP resource estimation:
  • Improved MMAP-related code in column.h (#36521)
  • Refined resource estimation when loading collections (#36728)
• Performance Enhancements:
  • Improved expression parsing efficiency by converting Unicode to ASCII (#36676)
  • Enabled parallel production of messages for multiple topics (#36462)
  • Reduced CPU overhead when calculating index file size (#36580)
  • Retrieved message type from header to minimize unmarshalling (#36454)
  • Optimized workload-based replica selection policy (#36384)
• Split delete task messages to fit within max message size limits (#36574)
• Added new RESTful URL to describe import jobs (#36754)
• Optimized import scheduling and added a time cost metric (#36684)
• Added balance report log for query coordinator balancer (#36749)
• Switched to using common GC configuration (#36670)
• Added streaming forward policy switch for delegator (#36712)
• Enabled manual compaction for collections without indexes (#36581)
• Enabled load balancing on query nodes with varying memory capacities (#36625)
• Unified case for inbound labels using metrics.label (#36616)
• Made transfer channel/segment operations idempotent (#36552)
• Added metrics to monitor import throughput and imported row count (#36588)
• Prevented creation of multiple timer objects in targets (#36573)
• Updated expression version and formatted HTTP response for expressions (#36467)
• Enhanced garbage collection in snapshot KV (#36793)
• Added support to execute methods with context parameters (#36798)

milvus-2.3.22

v2.3.22

Release date: October 9, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.22	2.3.7	2.3.6	2.3.6	2.3.5

Milvus v2.3.22 introduces several improvements and bug fixes, including:

Features

• [cherry-pick] changes to propagate traceid from client (#34640)

Bug fixes

• Fix a security vulnerability (#36155)
• Fix proxy panic after getimportstate failed (#35541)

milvus-2.4.12

v2.4.12

Release Date: September 26, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.12	2.4.7	2.4.4	2.4.9

Milvus 2.4.12 introduces significant enhancements and critical bug fixes. This release addresses data duplication issues and improves failure recovery speed, particularly when handling large numbers of deletions. However, a known issue persists where failure recovery can be slow when deleting massive amounts of data. We are actively working on resolving this issue.

Improvements

• Implemented graceful stop for flowgraph manager (#36358)
• Disabled index checks for non-loaded vector fields (#36280)
• Filtered out non-hit delete records during delta loading (#36272)
• Improved error handling for std::stoi exceptions (#36296)
• Disallowed keywords as field names or dynamic field names (#36108)
• Added metrics for delete entries in L0 segments (#36227)
• Implemented L0 forward policy to support remote loading (#36208)
• Added ANN field loading check in proxy (#36194)
• Enabled empty sparse row support (#36061)
• Fixed a security vulnerability (#36156)
• Implemented stats handler for request/response size metrics (#36118)
• Corrected size estimation for encoded array data (#36379)

Bug fixes

• Resolved metric type errors for collections with two vector fields (#36473)
• Fixed long buffering issues causing message queue reception failures (#36425)
• Implemented proper compact-to-segments return after split support (#36429)
• Resolved data race issues with node ID check goroutine (#36377)
• Removed element type check (#36324)
• Fixed concurrent access issues for growing and sealed segments (#36288)
• Implemented future stateful lock (#36333)
• Corrected offset usage in HybridSearch (#36287, #36253)
• Resolved dirty segment/channel leaks on QueryNode (#36259)
• Fixed primary key duplication handling (#36274)
• Enforced metric type setting in search requests (#36279)
• Fixed stored_index_files_size metric clearing issue (#36161)
• Corrected readwrite privilege group behavior for global API access (#36145)

milvus-2.4.11

v2.4.11

Release Date: September 11, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.11	2.4.6	2.4.3	2.4.8

Milvus 2.4.11 is a bug-fix release that addresses multiple critical issues related to the MARISA trie index, compaction, and loading operations. This release introduces new features to view expressions and improve delete stability. We recommend all users of the 2.4.x series to upgrade to this version to benefit from these improvements and fixes.

Features

• Added static view for expressions in 2.4 (#35954)
• Implemented delete buffer related quota logic (#35997)

Bug fixes

• Resolved Trie index range operation issue for GreaterThan and GreaterThanEqual comparisons (#36126)
• Corrected marisa_label_order usage in Trie index construction (#36060)
• Enhanced value checking for trie.predictive_search (#35999)
• Enabled Binary arithmetic expression support on inverted index (#36097)
• Fixed segment fault caused by Skipindex (#35908)
• Resolved memory leak in proxy meta cache (#36076)
• Renamed mmap file path to prevent directory conflicts (#35975)
• Improved logging and cleanup for failed/timeout tasks in mix compaction (#35967)
• Addressed logic deadlock during high memory usage by delegator (#36066)
• Implemented empty segment creation when compaction deletes all inserts (#36045)
• Corrected load field list population from old version load info in 2.4 (#36018)
• Fixed tracing config update logic in 2.4 (#35998)
• Resolved search/query request failures during dynamic partition release (#36019)
• Prevented override of fallback parameters (#36006)
• Ensured proper registration of privilege groups for validation (#35938)
• Prevented mistaken cleanup of db limiter nodes (#35992)
• Addressed issue with replicas not participating in queries after failure recovery (#35925)
• Resolved data race in clustering compaction writer (#35958)
• Fixed variable reference after move operation (#35904)
• Implemented clustering key skip load behavior check (#35899)
• Ensured single startup of querycoord observers in 2.4 (#35817)

Improvements

• Upgraded Milvus & proto version to 2.4.11 (#36069)
• Addressed memory leak in unit tests and enable use_asan option for unittest builds (#35857)
• Adjusted l0segmentsrowcount limits to more appropriate values (#36015)
• Modified deltalog memory estimation factor to one (#36035)
• Implemented slicesetequal for load field list comparisons (#36062)
• Reduced log frequency for delete operations (#35981)
• Upgraded etcd version to 3.5.14 (#35977)
• Optimized mmap-rss reduction after warmup (#35965)
• Removed cooling off period in rate limiter for read requests (#35936)
• Enhanced load field checking for previously loaded collections (#35910)
• Added support for dropping roles related to privilege lists in 2.4 (#35863)
• Implemented depguard rules to prohibit deprecated proto library usage (#35818)

Others

• Updated Knowhere version (#36067)

milvus-2.4.10

v2.4.10

Release Date: August 30, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.10	2.4.6	2.4.3	2.4.6

Milvus 2.4.10 introduces significant improvements in functionality and stability. Key features include support for upsert operations on AutoID-enabled collections, partial collection loading capabilities, and various memory-mapped (MMAP) configurations to optimize memory usage. This release also addresses several bugs causing panics, core dumps, and resource leaks. We recommend upgrading to take full advantage of these improvements.

Features

• Upsert with Auto ID: Support for upsert operations with automatic ID generation (#34633)
• Field Partial Load Collection [Beta Preview]: Allows loading specific fields of a collection (#35696)
• RBAC Enhancements:
  • Added RBAC message support for Change Data Capture (CDC) (#35562)
  • Introduced readonly/readwrite/admin privilege groups to simplify RBAC grant process (#35543)
  • New API for backing up and restoring RBAC configurations (#35513)
  • Refresh proxy cache after restoring RBAC metadata (#35636)
• Improved MMAP Configuration: More general configuration options to control MMAP behavior (#35609)
• Database Access Restrictions: New properties to restrict read access to databases (#35754)

Bug fixes

• Fixed Arrow Go client don't return error issue (#35820)
• Corrected inaccurate rate limiting (#35700)
• Resolved proxy panic after import-related API failures (#35559)
• Fixed potential mistaken deletions during GC channel checkpoints (#35708)
• Addressed panic due to empty candidate import segments (#35674)
• Corrected mmap memory deallocation (#35726)
• Ensured proper channel watching for upgrades from 2.2 to 2.4 (#35695)
• Fixed DataNode unwatching channel release function (#35657)
• Corrected partition count in RootCoord metadata (#35601)
• Resolved issues with dynamic config updates for certain parameters (#35637)

Improvements

Performance

• Optimized retrieval on dynamic fields (#35602)
• Improved bitset performance for AVX512 (#35480)
• Re-read value after once initialization for better efficiency (#35643)

Rolling upgrade improvements

• Marked query node as read-only after suspended (#35586)
• Prevented coexistence of old coordinator with new node/proxy (#35760)

Others

• Optimized Milvus core building process (#35660)
• Updated to protobuf-go v2 (#35555)
• Enhanced tracing with hex string encoding for traceid and spanid (#35568)
• Added hit segment number metrics for query hook (#35619)
• Improved compatibility with old SDK for configure load param feature (#35573)
• Added support for HTTP v1/v2 throttling (#35504)
• Fixed index memory estimation (#35670)
• Ability to write multiple segments in mix compactor to avoid large segment generation (#35648)

milvus-2.4.9

v2.4.9

Release Date: August 20, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.9	2.4.5	2.4.3	2.4.4

Milvus v2.4.9 addresses a critical issue which could return results less than limit (topk) in some corner cases and includes several key improvements to enhance the performance and usability of the platform.

Critical fixes

• Excluded l0 segment from readable snapshot (#35510).

Improvements

• Removed duplicated schema helper creation in the proxy (#35502).
• Added support for compiling Milvus on Ubuntu 20.04 (#35457).
• Optimized the use of locks and avoided double flush clustering buffer writer (#35490).
• Removed the invalid log (#35473).
• Added a clustering compaction user guide doc (#35428).
• Added support for dynamic fields in the schema helper (#35469).
• Added the msgchannel section in the generated YAML (#35466).

milvus-2.4.8

v2.4.8

Release Date: August 14, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.8	2.4.5	2.4.3	2.4.4

Milvus 2.4.8 introduced several significant improvements to the system's performance and stability. The most notable feature was the implementation of clustering compaction, a mechanism that enhances search and query efficiency by redistributing data in large collections based on a designated clustering key, reducing the amount of data scanned. Compaction was also decoupled from the shard DataNode, allowing any DataNode to perform compaction independently, which improved fault tolerance, stability, performance, and scalability. Additionally, the interface between the Go and C++ components was refactored to use asynchronous CGO calls, addressing issues like session timeouts, while several other performance optimizations were made based on profiling. The application dependencies were also updated to address known security vulnerabilities. Moreover, this release also includes numerous performance optimizations and critical bug fixes.

Features

• Implemented clustering compaction, allowing data to be redistributed based on a designated clustering key to enhance query efficiency (#34326), (#34363).

Improvements

• Implemented asynchronous search and retrieval capabilities in CGO. (#34200)
• Separated the compaction process from the Shard DataNode to improve system modularity. (#34157)
• Added support for client pooling in QueryNode within the proxy/delegator to enhance performance. (#35195)
• Integrated Sonic to minimize CPU overhead during JSON marshaling and unmarshaling in Gin and RestfulV1 handlers. (#35018)
• Introduced an in-memory cache to optimize authentication result retrieval. (#35272)
• Modified the default metric type for autoindex. [#34277, #34479]
• Refactored the runtime memory format for variable columns, leading to reduced memory usage. [#34367, #35012, #35041]
• Refactored compaction processes to enable persistent data storage. (#34268)
• Enabled memory-mapped file support for growing segments, improving memory management. (#34110)
• Improved access logs by adding RESTful API support, logging consistency levels, and distinguishing between system and user errors. [#34295, #34352, #34396]
• Enabled the configuration of replica numbers and resource groups at the cluster, database, and collection levels, with support for dynamic modifications. [#34561, #33981, #35293]
• Utilized the new range_search_k parameter in Knowhere to speed up range searches. (#34709)
• Applied blocked Bloom filters to enhance the speed of filter construction and querying. [#34377, #34922]
• Memory Usage Improvements:
  • Pre-allocated space for DataNode insert buffers. (#34205)
  • Pre-allocated FieldData for Reduce operations. (#34254)
  • Released records in delete codec to prevent memory leaks. (#34506)
  • Controlled concurrency level of the disk file manager during file loading. (#35282)
  • Optimized Go runtime garbage collection logic for timely memory release. (#34950)
  • Implemented a new seal policy for growing segments. (#34779)
• DataCoord Enhancements:
  • Reduced CPU usage. [#34231, #34309]
  • Implemented faster garbage collection exit logic. (#35051)
  • Improved worker node scheduling algorithms. (#34382)
  • Enhanced segment size control algorithm specifically for import operations. (#35149)
• Load Balancing Algorithm Improvements:
  • Reduced the memory overload factor on the delegator. (#35164)
  • Allocated fixed memory size for the delegator. (#34600)
  • Avoided excessive allocation of segments and channels for new query nodes. (#34245)
  • Reduced the number of tasks per scheduling cycle by Query Coordinator while increasing scheduling frequency. (#34987)
  • Enhanced channel balancing algorithm on the DataNode. (#35033)
• Expanded System Metrics: Added new metrics across various components to monitor specific aspects including:
  • Force-deny-writing state. (#34989)
  • Queue latency. (#34788)
  • Disk quota. (#35306)
  • Task execution time. (#35141)
  • Binlog size. (#35235)
  • Insert rate. (#35188)
  • Memory high water level. (#35188)
  • RESTful API metrics. (#35083)
  • Search latency. (#34783)

Changes

• For open-source users, this version changes the metric types in AutoIndex for FloatVector and BinaryVector to Cosine and Hamming, respectively.

• Fixed Third-Party Dependency Versions:

  • This release introduces fixed versions for certain third-party dependency libraries, significantly enhancing Milvus's software supply chain management.
  • By isolating the project from upstream changes, it safeguards daily builds from potential disruptions.
  • The update ensures stability by exclusively hosting validated C++ third-party packages on JFrog Cloud and utilizing Conan Recipe Revisions (RREV).
  • This approach mitigates the risk of breaking changes from updates in ConanCenter.
  • Developers using Ubuntu 22.04 will benefit immediately from these changes. However, developers on other operating systems may need to upgrade their glibc version to avoid compatibility issues.

Critical bug fixes

• Fixed an issue where deletion data was lost due to segments being omitted during L0 compaction. [#33980, #34363]
• Rectified a problem where delete messages failed to be forwarded due to incorrect data scope handling. (#35313)
• Resolved a SIGBUS exception that occurred due to incorrect usage of mmap. [#34455, #34530]
• Fixed crashes caused by illegal search expressions. (#35307)
• Corrected an issue where DataNode watch failed due to an incorrect timeout setting in the watch context. (#35017)

Bug fixes

• Addressed security vulnerabilities by upgrading certain dependencies. [#33927, #34693]
• Fixed a parsing error triggered by excessively long expressions. (#34957)
• Resolved a memory leak that occurred during query plan parsing. (#34932)
• Fixed an issue where dynamic log level modifications were not taking effect. (#34777)
• Resolved an issue where group by queries on growing data failed due to uninitialized segment offsets. (#34750)
• Corrected the setting of search parameters when using the Knowhere iterator. (#34732)
• Revised the logic for checking the status of the partition load. (#34305)
• Fixed an issue where privilege cache updates failed due to unhandled request errors. (#34697)
• Resolved a failure in loaded collection recovery after QueryCoord restarted. (#35211)
• Fixed a...

milvus-2.3.21

v2.3.21

Release data: August 14, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.21	2.3.7	2.3.6	2.3.6	2.3.5

Miluvs v2.3.21 addresses a critical bug in data deletion (#35390) and introduces some improvements in metrics and performance.

Improvements

• Enhanced data deletion process by modifying the delegator's delete buffer handling (#35074)
• Added metrics to track disk quota usage (#35320)
• Reduced delegator memory usage by adjusting the overloaded factor to 0.1 (#35165)
• Improved health check by skipping manually stopped components (#35122)
• Added metric to record maxinsertrate and querynodememoryhighwaterlevel (#35193)
• Upgraded the Conan version (#35217)
• Changed the default metric type for autoindex (#34328)
• Sped up the datacoord gc quit process (#35057)
• Avoided panic due to nil schema (#35065)
• Reduced duplicate primary keys in the segcore component (#35291)
• Tracked the number of times Milvus enters a force-deny-writing state (#34990)

Bug fixes

• Resolved an issue with the BloomFilter returning false negatives when the statslog contained multiple K values, which could occur when a delete operation failed to apply (#35390)
• Enabled a limiter for RESTful server (#35354)
• Added a retry mechanism to avoid getting incomplete query results (#35061)
• Fixed issues with delete-by-expression operations failing during retry progress (#35421)
• Addressed a problem where the datanode binlog size metrics included dropped segments (#35420)
• Resolved comparison operations between incompatible operands (#35308)
• Fixed an issue where the datanode could not watch the channel (#35136)
• Addressed a deadlock issue on compaction when stopping the datanode (#35199)

milvus-2.3.20

v2.3.20

Release data: July 25, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.20	2.3.7	2.3.6	2.3.6	2.3.5

Milvus v2.3.20 brings several important bug fixes focused on enhancing segment management. This release also includes an upgrade to Go version 1.21, which allows the pprof tool to capture C++ code profiles, significantly aiding in diagnosing Milvus. Additionally, the Knowhere version has been updated to v2.2.7, enabling Milvus to be compiled with glibc versions ≤ 2.30. A new metric has been introduced to record the duration that requests wait in the proxy queue, improving Milvus's observability.

Features

• Added a metric to record the duration of requests waiting in the proxy queue (#34791).

Improvements

• Upgraded Knowhere to version 2.2.7 (#34825).
• Upgraded Go version from 1.20 to 1.21 (#34759).

Bug fixes

• Initialized the hook when creating the proxy object (#34936).
• Fixed a parsing issue for the plan proto for search types (#34945).
• Resolved an issue where segments could bounce between delegator and worker (#34889).
• Prevented the generation of reduce channel tasks before channel balancing is complete (#34721).
• Avoided segment shortages caused by deduplicated segment tasks (#34882).
• Discarded compaction plans when closing DataSyncService (#34737).
• Fixed a typo in the compatible ascending index (#34712).