Manticore Search 6.3.8

Manticore Search 6.3.8

Released: November 22nd 2024

➡️ INSTALL ⬅️

Version 6.3.8 continues the 6.3 series and includes only bug fixes.

Bug fixes

• PR #2777 Fixed the calculation of available threads when query concurrency is limited by the threads or max_threads_per_query settings.

Manticore Search 6.3.6

Manticore Search 6.3.6

Released: August 2nd 2024

➡️ INSTALL ⬅️

Version 6.3.6 continues the 6.3 series and includes only bug fixes.

Bug fixes

• Issue #2477 Fixed a crash introduced in version 6.3.4, which could occur when dealing with expressions and distributed or multiple tables.
• Issue #2352 Fixed a daemon crash or internal error upon early exit caused by max_query_time when querying multiple indexes.

Manticore Search 6.3.4

Version 6.3.4

Released: July 31st 2024

➡️ INSTALL ⬅️

Version 6.3.4 continues the 6.3 series and includes only minor improvements and bug fixes.

Minor changes

• Issue #1130 Added support for DBeaver.
• Issue #2146 Improved escaping of delimiters in word forms and exceptions.
• Issue #2268 Improved external files renaming on copy for CREATE and ALTER TABLE statements.
• Issue #2315 Added string comparison operators to SELECT list expressions.
• Issue #2363 Added support for null values in Elastic-like bulk requests.
• Issue #2374 Added support for mysqldump version 9.
• Issue #2375 Improved error handling in HTTP JSON queries with JSON path to the node where the error occurs.

Bug fixes

• Issue #2280 Fixed performance degradation in wildcard queries with many matches when disk_chunks > 1.
• Issue #2332 Fixed crash in MVA MIN or MAX SELECT list expressions for empty MVA arrays.
• Issue #2339 Fixed incorrect processing of Kibana's infinite range request.
• Issue #2342 Fixed join filter on columnar attributes from the right table when the attribute is not in the SELECT list.
• Issue #2343 Fixed duplicate 'static' specifier in Manticore 6.3.2.
• Issue #2344 Fixed LEFT JOIN returning non-matching entries when MATCH() over the right table is used.
• Issue #2350 Fixed saving of disk chunk at RT index with hitless_words.
• Issue #2364 The 'aggs_node_sort' property can now be added in any order among other properties.
• Issue #2368 Fixed error on full-text vs filter order in the JSON query.
• Issue #2376 Fixed bug related to incorrect JSON response for long UTF-8 requests.
• Issue #2684 Fixed calculation of presort/prefilter expressions that depend on joined attributes.
• Issue #301 Changed the method of calculating data size for metrics to read from the manticore.json file instead of checking the entire size of the data directory.
• Issue #302 Added handling of validation requests from Vector.dev.

Manticore Search 6.3.2

Version 6.3.2

Released: June 26th 2024

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Version 6.3.2 continues the 6.3 series and includes several bug fixes, some of which were discovered after the release of 6.3.0.

Breaking changes

• ⚠️Issue #2305 Updated aggs.range values to be numeric.

Bug fixes

• Commit c51c Fixed grouping by stored check vs rset merge.
• Commit 0e85 Fixed a crash in the daemon when querying with wildcard characters in an RT index using a CRC dictionary and local_df enabled.
• Issue #2200 Fixed a crash in JOIN on count(*) without GROUP BY.
• Issue #2201 Fixed JOIN not returning a warning when attempting grouping by a full-text field.
• Issue #2230 Addressed issue where adding an attribute via ALTER TABLE did not take KNN options into account.
• Issue #2231 Fixed failure in removing manticore-tools Redhat package in version 6.3.0.
• Issue #2242 Corrected issues with JOIN and multiple FACET statements returning incorrect results.
• Issue #2250 Fixed ALTER TABLE producing an error if the table is in a cluster.
• Issue #2252 Fixed the original query being passed into buddy from the SphinxQL interface.
• Issue #2267 Improved wildcard expansion in the CALL KEYWORDS for RT index with disk chunks.
• Issue #271 Fixed hanging of incorrect /cli requests.
• Issue #274 Resolved issues where concurrent requests to Manticore could get stuck.
• Issue #275 Fixed hanging of drop table if exists t; create table t via /cli.

Replication-related

• Issue #2270 Added support for cluster:name format in the /_bulk HTTP endpoint.

Manticore Search 6.3.0

Version 6.3.0

Released: May 23rd 2024

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

📢📢📢Announcement blog post📢📢📢

Major changes

• Issue #839 Implemented float_vector data type; implemented vector search.
• Issue #1673 INNER/LEFT JOIN (beta stage).
• Issue #1744 Implemented autodetection of date formats for timestamp fields.
• Issue #1720 Changed Manticore Search license from GPLv2-or-later to GPLv3-or-later.
• Commit 7a55 Running Manticore in Windows now requires Docker to run Buddy.
• Issue #1541 Added a REGEX full-text operator.
• Issue #2091 Ubuntu Noble 24.04 support.
• Commit 514d Revamp of time operations for better performance and new date/time functions:
  • CURDATE() - Returns current date in local timezone
  • QUARTER() - Returns the integer quarter of the year from a timestamp argument
  • DAYNAME() - Returns the weekday name for a given timestamp argument
  • MONTHNAME() - Returns the name of the month for a given timestamp argument
  • DAYOFWEEK() - Returns the integer weekday index for a given timestamp argument
  • DAYOFYEAR() - Returns the integer day of the year for a given timestamp argument
  • YEARWEEK() - Returns the integer year and the day code of the first day of current week for a given timestamp argument
  • DATEDIFF() - Returns the number of days between two given timestamps
  • DATE() - Formats the date part from a timestamp argument
  • TIME() - Formats the time part from a timestamp argument
  • timezone - Timezone used by date/time-related functions.
• Commit 30e7 Added range, histogram, date_range, and date_histogram aggregates to the HTTP interface and similar expressions into SQL.

Minor changes

• Issue #1285 Support of Filebeat versions 8.10 - 8.11.
• Issue #1771 ALTER TABLE ... type='distributed'.
• Issue #1788 Added the ability to copy tables using the CREATE TABLE ... LIKE ... WITH DATA SQL statement.
• Issue #2072 Optimized the table compacting algorithm: Previously, both manual OPTIMIZE and automatic auto_optimize processes would first merge chunks to ensure the count did not exceed the limit, and then expunge deleted documents from all other chunks containing deleted documents. This approach was sometimes too resource-intensive and has been disabled. Now, chunk merging occurs solely according to the progressive_merge setting. However, chunks with a high number of deleted documents are more likely to be merged.
• Commit ce6c Added protection against loading a secondary index of a newer version.
• Issue #1417 Partial replace via REPLACE INTO ... SET.
• Commit 7c16 Updated default merge buffer sizes: .spa (scalar attrs): 256KB -> 8MB; .spb (blob attrs): 256KB -> 8MB; .spc (columnar attrs): 1MB, no change; .spds (docstore): 256KB -> 8MB; .spidx (secondary indexes): 256KB buffer -> 128MB memory limit; .spi (dictionary): 256KB -> 16MB; .spd (doclists): 8MB, no change; .spp (hitlists): 8MB, no change; .spe (skiplists): 256KB -> 8MB.
• Issue #1859 Added composite aggregation via JSON.
• Commit 216b Disabled PCRE.JIT due to issues with some regex patterns and no significant time benefit.
• Commit 55cd Added support for vanilla Galera v.3 (api v25) (libgalera_smm.so from MySQL 5.x).
• Commit 86f9 Changed metric suffix from _rate to _rps.
• Commit c0c1 Improved docs about balancer HA support.
• Commit d1d2 Changed index to table in error messages; fixed bison parser error message fixup.
• Commit fd26 Support manticore.tbl as table name.
• Issue #1105 Support for running indexer via systemd (docs). ❤️ Thank you, @subnix for the PR.
• Issue #1294 Secondary indexes support in GEODIST().
• Issue #1394 Simplified SHOW THREADS.
• Issue #1424 Added support for the default values (agent_connect_timeout and agent_query_timeout) for create distributed table statement.
• Issue #1442 Added expansion_limit search query option that overrides searchd.expansion_limit.
• Issue #1448 Implemented ALTER TABLE for int->bigint conversion.
• Issue #146 Meta information in MySQL response.
• Issue #1494 SHOW VERSION.
• Issue #1582 Support of deleting documents by id array via JSON.
• Issue #1589 Improve error "unsupported value type".
• Issue #1634 Added Buddy ve...

Manticore Search 6.2.12

Manticore Search 6.2.12

Released: Aug 23rd 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Version 6.2.12 continues the 6.2 series and addresses issues discovered after the release of 6.2.0.

Bugfixes

• ❗Issue #1351 "Manticore 6.2.0 doesn't start via systemctl on Centos 7": Modified TimeoutStartSec from infinity to 0 for better compatibility with Centos 7.
• ❗Issue #1364 "Crash after upgrading from 6.0.4 to 6.2.0": Added replay functionality for empty binlog files from older binlog versions.
• PR #1334 "fix typo in searchdreplication.cpp": Corrected a typo in searchdreplication.cpp: beggining -> beginning.
• Issue #1337 "Manticore 6.2.0 WARNING: conn (local)(12), sock=8088: bailing on failed MySQL header, AsyncNetInputBuffer_c::AppendData: error 11 (Resource temporarily unavailable) return -1": Lowered the verbosity level of the MySQL interface warning about the header to logdebugv.
• Issue #1355 "join cluster hangs when node_address can't be resolved": Improved replication retry when certain nodes are unreachable, and their name resolution fails. This should resolve issues in Kubernetes and Docker nodes related to replication. Enhanced the error message for replication start failures and made updates to test model 376. Additionally, provided a clear error message for name resolution failures.
• Issue #1361 "No lower case mapping for "Ø" in charset non_cjk": Adjusted the mapping for the 'Ø' character.
• Issue #1365 "searchd leaves binlog.meta and binlog.001 after clean stop": Ensured that the last empty binlog file is removed properly.
• Commit 0871: Fixed the Thd_t build issue on Windows related to atomic copy restrictions.
• Commit 1cc0: Addressed an issue with FT CBO vs ColumnarScan.
• Commit c6bf: Made corrections to test 376 and added a substitution for the AF_INET error in the test.
• Commit cbc3: Resolved a deadlock issue during replication when updating blob attributes versus replacing documents. Also removed the rlock of the index during commit because it's already locked at a more basic level.

Minor changes

• Commit 4f91 Updated info on /bulk endpoints in the manual.

MCL

• Support of Manticore Columnar Library v2.2.4

Manticore Search 6.2.0

Manticore Search 6.2.0

Released: Aug 4th 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Major changes

• The query optimizer has been enhanced to support full-text queries, significantly improving search efficiency and performance.
• Integrations with:
  • mysqldump - to make logical backups using mysqldump
  • Apache Superset and Grafana to visualize data stored in Manticore
  • HeidiSQL and DBForge for easier development with Manticore
• We've started using GitHub workflows, making it simpler for contributors to utilize the same Continuous Integration (CI) process that the core team applies when preparing packages. All jobs can be run on GitHub-hosted runners, which facilitates seamless testing of changes in your fork of Manticore Search.
• We've started using CLT to test complex scenarios. For example, we're now able to ensure that a package built after a commit can be properly installed across all supported Linux operating systems. The Command Line Tester (CLT) provides a user-friendly way to record tests in an interactive mode and to effortlessly replay them.
• Significant performance improvement in count distinct operation by employing a combination of hash tables and HyperLogLog.
• Enabled multithreaded execution of queries containing secondary indexes, with the number of threads limited to the count of physical CPU cores. This should considerably improve the query execution speed.
• pseudo_sharding has been adjusted to be limited to the number of free threads. This update considerably enhances the throughput performance.
• Users now have the option to specify the default attribute storage engine via the configuration settings, providing better customization to match specific workload requirements.
• Support for Manticore Columnar Library 2.2.0 with numerous bug fixes and improvements in Secondary indexes.

Minor changes

• Buddy #153: The /pq HTTP endpoint now serves as an alias for the /json/pq HTTP endpoint.
• Commit 0bf1: We've ensured multi-byte compatibility for upper() and lower().
• Commit 2bb9: Instead of scanning the index for count(*) queries, a precalculated value is now returned.
• Commit 3c84: It's now possible to use SELECT for making arbitrary calculations and displaying @@sysvars. Unlike before, you are no longer limited to just one calculation. Therefore, queries like select user(), database(), @@version_comment, version(), 1+1 as a limit 10 will return all the columns. Note that the optional 'limit' will always be ignored.
• Commit 6aca: Implemented the CREATE DATABASE stub query.
• Commit 9dc1: When executing ALTER TABLE table REBUILD SECONDARY, secondary indexes are now always rebuilt, even if attributes weren't updated.
• Commit 46ed: Sorters utilizing precalculated data are now identified before using CBO to avoid unnecessary CBO calculations.
• Commit 102a: Implementing mocked and utilizing of the full-text expression stack to prevent daemon crashes.
• Commit 979f: A speedy code path has been added for match cloning code for matches that don't use string/mvas/json attributes.
• Commit a073: Added support for the SELECT DATABASE() command. However, it will always return Manticore. This addition is crucial for integrations with various MySQL tools.
• Commit bc04: Modified the response format of the /cli endpoint, and added the /cli_json endpoint to function as the previous /cli.
• Commit d70b: The thread_stack can now be altered during runtime using the SET statement. Both session-local and daemon-wide variants are available. Current values can be accessed in the show variables output.
• Commit d96e: Code has been integrated into CBO to more accurately estimate the complexity of executing filters over string attributes.
• Commit e77d: The DocidIndex cost calculation has been improved, enhancing overall performance.
• Commit f3ae: Load metrics, similar to 'uptime' on Linux, are now visible in the SHOW STATUS command.
• Commit f3cc: The field and attribute order for DESC and SHOW CREATE TABLE now match that of SELECT * FROM.
• Commit f3d2: Different internal parsers now provide their internal mnemonic code (e.g., P01) during various errors. This enhancement aids in identifying which parser caused an error and also obscures non-essential internal details.
• Issue #271 "Sometimes CALL SUGGEST does not suggest a correction of a single letter typo": Improved SUGGEST/QSUGGEST behaviour for short words: added the option sentence to show the entire sentence
• Issue #696 "Percolate index does not search properly by exact phrase query when stemming enabled": The percolate query has been modified to handle an exact term modifier, improving search functionality.
• Issue #829 "DATE FORMATTING methods": added the date_format() select list expression, which exposes the strftime() function.
• Issue #961 "Sorting buckets via HTTP JSON API": introduced an optional sort property for each bucket of aggregates in the HTTP interface.
• Issue #1062 "Improve error logging of JSON insert api failure - "unsupported value type"": The /bulk endpoint reports information regarding the number of processed and non-processed strings (documents) in case of an error.
• Issue #1070 "CBO hints don't support multiple attributes": Enabled index hints to handle multiple attributes.
• Issue #1106 "Add tags to http search query": Tags have been added to HTTP PQ responses.
• Issue #1301 "buddy should not create table in parallel": Resolved an issue that was causing failures from parallel CREATE TABLE operations. Now, only one CREATE TABLE operation can run at a time.
• Issue #1303 "add support of @ to column names".
• Issue #1316 "Queries on taxi dataset are slow with ps=1": The CBO logic has been refined, and the default histogram resolution has been set to 8k for better accuracy on attributes with randomly distributed values.
• Issue #1317 "Fix CBO vs fulltext on hn dataset": Enhanced logic has been implemented for determining when to use bitmap iterator intersection and when to use a priority queue.
• Issue #1318 "columnar: change iterator interface to single-call" : Columnar iterators now use a single Get call, replacing the previous two-step AdvanceTo + Get calls to retrieve a value.
• Issue #1319 "Aggregate calc speedup (remove CheckReplaceEntry?)": The CheckReplaceEntry call was removed from...

Manticore Search 6.0.4

Manticore Search 6.0.4

Released: Mar 15 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

New features

• Improved integration with Logstash, Beats etc. including:
  • Support for Logstash versions >= 7.13.
  • Auto-schema support.
  • Added handling of bulk requests in Elasticsearch-like format.
• Buddy commit ce90 Log Buddy version on Manticore start.

Bugfixes

• Issue #588, Issue #942 Fixed bad character at the search meta and call keywords for bigram index.
• Issue #1027 Lowercase HTTP headers are rejected.
• ❗Issue #1039 Fixed memory leak at daemon on reading output of the Buddy console.
• Issue #1056 Fixed unexpected behavior of question mark.
• Issue #1064 - Fixed race condition in tokenizer lowercase tables causing a crash.
• Commit 59bb Fixed bulk writes processing in the JSON interface for documents with id explicitly set to null.
• Commit 7b6b Fixed term statistics in CALL KEYWORDS for multiple same terms.
• Commit f381 Default config is now created by Windows installer; paths are no longer substituted in runtime.
• Commit 6940, Commit cc5a Fixed replication issues for cluster with nodes in multiple networks.
• Commit 4972 Fixed /pq HTTP endpoint to be an alias of the /json/pq HTTP endpoint.
• Commit 3b53 Fixed daemon crash on Buddy restart.
• Buddy commit fba9 Display original error on invalid request received.
• Buddy commit db95 Allow spaces in backup path and add some magic to regexp to support single quotes also.

Manticore Search 6.0.2

Manticore Search 6.0.2

Released: Feb 10 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Bugfixes

• Issue #1024 crash 2 Crash / Segmentation Fault on Facet search with larger number of results
• ❗Issue #1029 - WARNING: Compiled-in value KNOWN_CREATE_SIZE (16) is less than measured (208). Consider to fix the value!
• ❗Issue #1032 - Manticore 6.0.0 plain index crashes
• ❗Issue #1033 - multiple distributed lost on daemon restart

Manticore Search 6.0.0

Manticore Search 6.0.0

Released: Feb 7 2023

➡️➡️➡️ DOWNLOAD HERE ⬅️⬅️⬅️

Starting with this release, Manticore Search comes with Manticore Buddy, a sidecar daemon written in PHP that handles high-level functionality that does not require super low latency or high throughput. Manticore Buddy operates behind the scenes, and you may not even realize it is running. Although it is invisible to the end user, it was a significant challenge to make Manticore Buddy easily installable and compatible with the main C++-based daemon. This major change will allow the team to develop a wide range of new high-level features, such as shards orchestration, access control and authentication, and various integrations like mysqldump, DBeaver, Grafana mysql connector. For now it already handles SHOW QUERIES, BACKUP and Auto schema.

This release also includes more than 130 bug fixes and numerous features, many of which can be considered major.

Major Changes

• 🔬 Experimental: you can now execute Elasticsearch-compatible insert and replace JSON queries which enables using Manticore with tools like Logstash (version < 7.13), Filebeat and other tools from the Beats family. Enabled by default. You can disable it using SET GLOBAL ES_COMPAT=off.
• Support for Manticore Columnar Library 2.0.0 with numerous fixes and improvements in Secondary indexes. ⚠️ BREAKING CHANGE: Secondary indexes are ON by default as of this release. Make sure you do ALTER TABLE table_name REBUILD SECONDARY if you are upgrading from Manticore 5. See below for more details.
• Commit c436 Auto-schema: you can now skip creating a table, just insert the first document and Manticore will create the table automatically based on its fields. Read more about this in detail here. You can turn it on/off using searchd.auto_schema.
• Vast revamp of cost-based optimizer which lowers query response time in many cases.
  • Issue #1008 Parallelization performance estimate in CBO.
  • Issue #1014 CBO is now aware of secondary indexes and can act smarter.
  • Commit cef9 Encoding stats of columnar tables/fields are now stored in the meta data to help CBO make smarter decisions.
  • Commit 2b95 Added CBO hints for fine-tuning its behaviour.
• Telemetry: we are excited to announce the addition of telemetry in this release. This feature allows us to collect anonymous and depersonalized metrics that will help us improve the performance and user experience of our product. Rest assured, all data collected is completely anonymous and will not be linked to any personal information. This feature can be easily turned off in the settings if desired.
• Commit 5aaf ALTER TABLE table_name REBUILD SECONDARY to rebuild secondary indexes whenever you want, for example:
  • when you migrate from Manticore 5 to the newer version,
  • when you did UPDATE (i.e. in-place update, not replace) of an attribute in the index
• Issue #821 New tool manticore-backup for backing up and restoring Manticore instance
• SQL command BACKUP to do backups from inside Manticore.
• SQL command SHOW QUERIES as an easy way to see running queries rather than threads.
• Issue #551 SQL command KILL to kill a long-running SELECT.
• Dynamic max_matches for aggregation queries to increase accuracy and lower response time.

Minor changes

• Issue #822 SQL commands FREEZE/UNFREEZE to prepare a real-time/plain table for a backup.

• Commit c470 New settings accurate_aggregation and max_matches_increase_threshold for controlled aggregation accuracy.

• Issue #718 Support for signed negative 64-bit IDs. Note, you still can't use IDs > 2^63, but you can now use ids in the range of from -2^63 to 0.

• As we recently added support for secondary indexes, things became confusing as "index" could refer to a secondary index, a full-text index, or a plain/real-time index. To reduce confusion, we are renaming the latter to "table". The following SQL/command line commands are affected by this change. Their old versions are deprecated, but still functional:

  • index <table name> => table <table name>,
  • searchd -i / --index => searchd -t / --table,
  • SHOW INDEX STATUS => SHOW TABLE STATUS,
  • SHOW INDEX SETTINGS => SHOW TABLE SETTINGS,
  • FLUSH RTINDEX => FLUSH TABLE,
  • OPTIMIZE INDEX => OPTIMIZE TABLE,
  • ATTACH TABLE plain TO RTINDEX rt => ATTACH TABLE plain TO TABLE rt,
  • RELOAD INDEX => RELOAD TABLE,
  • RELOAD INDEXES => RELOAD TABLES.

  We are not planning to make the old forms obsolete, but to ensure compatibility with the documentation, we recommend changing the names in your application. What will be changed in a future release is the "index" to "table" rename in the output of various SQL and JSON commands.

• Queries with stateful UDFs are now forced to be executed in a single thread.

• Issue #1011 Refactoring of all related to time scheduling as a prerequisite for parallel chunks merging.

• ⚠️ BREAKING CHANGE: Columnar storage format has been changed. You need to rebuild those tables that have columnar attributes.

• ⚠️ BREAKING CHANGE: Secondary indexes file format has been changed, so if you are using secondary indexes for searching and have searchd.secondary_indexes = 1 in your configuration file, be aware that the new Manticore version will skip loading the tables that have secondary indexes. It's recommended to:

  • Before you upgrade change searchd.secondary_indexes to 0 in the configuration file.
  • Run the instance. Manticore will load up the tables with a warning.
  • Run ALTER TABLE <table name> REBUILD SECONDARY for each index to rebuild secondary indexes.

  If you are running a replication cluster, you'll need to run ALTER TABLE <table name> REBUILD SECONDARY on all the nodes or follow this instruction with just change: run the ALTER .. REBUILD SECONDARY instead of the OPTIMIZE.

• ⚠️ BREAKING CHANGE: The binlog version has been updated, so any binlogs from previous versions will not be replayed. It is important to ensure that Manticore Search is stopped cleanly during the upgrade process. This means that there should be no binlog files in /var/lib/manticore/binlog/ except for binlog.meta after stopping the previous instance.

• Issue #849 SHOW SETTINGS: helper command for manticore-backup.

• Issue #1007 SET GLOBAL CPUSTATS=1/0 turns on/off cpu time tracking; SHOW THREADS now doesn'...