High Memory usage on Manticore instance

[mohdmsl]

Bug Description:

I have 7 columnar indexes but there is high unaccounted memory usage:
Image 1:

Below are the status of indexes using memory and memory mapped to disk. which is very low
Image 2:

Here if you see memory usage is of around 22gb but indexes are using way less memory. I am not sure where is this 20gb of memory being used.

Below is memory usage of manticore instance:

root@manticore-dev-01:/var/log/manticore# free -h
              total        used        free      shared  buff/cache   available
Mem:            30G         22G        3.3G        1.8M        4.9G        7.9G
Swap:            0B          0B          0B

Schema of my tables is:

CREATE TABLE IF NOT EXISTS lisdocumentchunk_jina_v3_onnx_fp16 (document_id string,
             source_document_id string,
             page_number bigint,
             chunk_index bigint,
             start_char bigint,
             end_char bigint,  
             chunk_len_chars bigint,
             chunk_text text stored indexed,
             vector float_vector  knn_type='hnsw' knn_dims=768' hnsw_similarity='COSINE')
stopwords = 'en' html_strip = '1' min_prefix_len='3' min_infix_len = '3' expand_keywords= '1' min_word_len = '3' min_stemming_len = '3' stopwords_unstemmed = '1' index_exact_words='1' blend_chars='+,&' rt_mem_limit='512M'  engine='columnar'

status of table:

show table lisdocumentchunk_jina_v3_onnx_fp16 status;
+-----------------------------+--------------------------------------------------------------------------+
| Variable_name               | Value                                                                    |
+-----------------------------+--------------------------------------------------------------------------+
| index_type                  | rt                                                                       |
| indexed_documents           | 1071708                                                                  |
| indexed_bytes               | 7761637318                                                               |
| ram_bytes                   | 69864                                                                    |
| disk_bytes                  | 11802761448                                                              |
| disk_mapped                 | 80900938                                                                 |
| disk_mapped_cached          | 0                                                                        |
| disk_mapped_doclists        | 0                                                                        |
| disk_mapped_cached_doclists | 0                                                                        |
| disk_mapped_hitlists        | 0                                                                        |
| disk_mapped_cached_hitlists | 0                                                                        |
| killed_documents            | 0                                                                        |
| killed_rate                 | 0.00%                                                                    |
| ram_chunk                   | 0                                                                        |
| ram_chunk_segments_count    | 0                                                                        |
| disk_chunks                 | 16                                                                       |
| mem_limit                   | 536870912                                                                |
| mem_limit_rate              | 33.33%                                                                   |
| ram_bytes_retired           | 0                                                                        |
| locked                      | 0                                                                        |
| tid                         | 0                                                                        |
| tid_saved                   | 0                                                                        |
| query_time_1min             | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_5min             | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_15min            | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_total            | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_1min             | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_5min             | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_15min            | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_total            | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
+-----------------------------+--------------------------------------------------------------------------+

Manticore Search Version:

6.3.6

Operating System Version:

linux

Have you tried the latest development version?

None

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated

[mohdmsl]

also

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mantico+ 26685  5.1 66.4 35552496 21415116 ?   Sl   Nov01 284:53 /usr/bin/searchd --config /etc/manticoresearch/manticore.conf
root       460  0.0  0.3 221452 116404 ?       S<s  Oct01   3:53 /lib/systemd/systemd-journald
root      1102  0.0  0.0 1854772 24708 ?       Ssl  Oct01   2:59 /usr/lib/snapd/snapd
root     20582  0.0  0.0 2434060 23704 ?       Sl   Oct26   3:57 /snap/amazon-ssm-agent/9881/ssm-agent-worker
root      1120  0.1  0.0 1805212 18876 ?       Ssl  Oct01  59:53 /usr/bin/containerd
root     20420  0.0  0.0 2129052 14924 ?       Ssl  Oct26   0:57 /snap/amazon-ssm-agent/9881/amazon-ssm-agent
mantico+ 28648  0.0  0.0  52972 13272 ?        S    Nov01   0:32 manticoresearch-buddy-worker [0]
mantico+ 28650  0.0  0.0  52896 12568 ?        S    Nov01   0:06 manticoresearch-buddy-worker [1]
mantico+ 28641  0.0  0.0  48732 11732 ?        S    Nov01   0:09 manticoresearch-buddy-manager

[sanikolaev]

Please use pmap or /proc/PID/maps / /proc/PID/smaps to learn more about RAM usage. They show details on mapped files, how much memory they take, and anonymous memory usage. If something looks wrong to you, please share the details here.