[Bug]: SIGSEGV in unit tests

[alexanderguzhva]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:
master branch as of Oct 17th (b474374ea568ea89333d5d963cf0095b151a53c0)
- Deployment mode(standalone or cluster):
local machine, built from the git source
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
ubuntu
- CPU/Memory: 
- GPU: 
- Others: 
clang-17

Current Behavior

Crashing unit tests

> CC=clang-17 CXX=clang++-17 make test-cpp

...

[----------] 19 tests from ExprMaterializedViewTest
[ RUN      ] ExprMaterializedViewTest.TestMvNoExpr
[       OK ] ExprMaterializedViewTest.TestMvNoExpr (1 ms)
[ RUN      ] ExprMaterializedViewTest.TestMvNotInvolvedExpr
[       OK ] ExprMaterializedViewTest.TestMvNotInvolvedExpr (1 ms)
[ RUN      ] ExprMaterializedViewTest.TestMvNotInvolvedJsonExpr
*** Aborted at 1729170400 (Unix time, try 'date -d @1729170400') ***
*** Signal 11 (SIGSEGV) (0x9) received by PID 213128 (pthread TID 0x772f2f111a80) (linux TID 213128) (code: address not mapped to object), stack trace: ***
(error retrieving stack trace)
/home/ubuntu/zilliz/milvus/milvus/scripts/run_cpp_unittest.sh: line 42: 213128 Segmentation fault      (core dumped) ${UNITTEST_DIR}/all_tests
make: *** [Makefile:358: test-cpp] Error 139

I don't believe that this is clang-specific behavior.

The bug is reproduceable

Expected Behavior

no SIGSEGV

Steps To Reproduce

1. ubuntu 22.04
2. `git clone https://github.com/milvus-io/milvus`
3. set up clang-17 as a compiler
4. run `make test-cpp`, Release build
5. get a SIGSEGV

Debug build does not lead to SIGSEGV.

Milvus Log

gdb log (release mode, so dumped values may be crazy)

[----------] 19 tests from ExprMaterializedViewTest
[ RUN      ] ExprMaterializedViewTest.TestMvNoExpr
[       OK ] ExprMaterializedViewTest.TestMvNoExpr (3 ms)
[ RUN      ] ExprMaterializedViewTest.TestMvNotInvolvedExpr
[       OK ] ExprMaterializedViewTest.TestMvNotInvolvedExpr (1 ms)
[ RUN      ] ExprMaterializedViewTest.TestMvNotInvolvedJsonExpr

Thread 1 "all_tests" received signal SIGSEGV, Segmentation fault.
simdjson::fallback::ondemand::json_iterator::last_position (this=0x7fffffffb210) at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/json_iterator-inl.h:280
280	  uint32_t n_structural_indexes{parser->implementation->n_structural_indexes};
(gdb) bt
#0  simdjson::fallback::ondemand::json_iterator::last_position (this=0x7fffffffb210)
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/json_iterator-inl.h:280
#1  simdjson::fallback::ondemand::json_iterator::peek_last (this=0x7fffffffb210)
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/json_iterator-inl.h:285
#2  simdjson::fallback::ondemand::value_iterator::check_root_object (this=<optimized out>)
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/value_iterator-inl.h:48
#3  simdjson::fallback::ondemand::value_iterator::started_root_object (this=<optimized out>)
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/value_iterator-inl.h:68
#4  simdjson::fallback::ondemand::value_iterator::start_root_object (this=<optimized out>)
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/value_iterator-inl.h:20
#5  simdjson::fallback::ondemand::object::start_root (iter=...)
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/object-inl.h:41
#6  simdjson::fallback::ondemand::document::get_object() & (this=0x7fffffffb210)
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/document-inl.h:83
#7  simdjson::fallback::ondemand::document::at_pointer (this=0x7fffffffb210, json_pointer="/A")
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/document-inl.h:272
#8  simdjson::simdjson_result<simdjson::fallback::ondemand::document>::at_pointer (this=0x7fffffffb210, json_pointer="/A")
    at /home/ubuntu/zilliz/milvus/milvus/cmake_build/3rdparty_download/simdjson-src/include/simdjson/generic/ondemand/document-inl.h:533
#9  milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long>()::{lambda(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&)#1}::operator()(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&) const::{lambda(unsigned long)#1}::operator()(unsigned long) const (this=this@entry=0x7fffffffb330, i=i@entry=0)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/JsonContainsExpr.cpp:240
#10 0x00007ffff47da520 in milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long>()::{lambda(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&)#1}::operator()(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&) const
    (data=0x7fffe18f8000, size=1000, res=..., pointer="/A", elements=..., this=<optimized out>)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/JsonContainsExpr.cpp:256
#11 milvus::exec::SegmentExpr::ProcessChunkForSealedSeg<milvus::Json, milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long>()::{lambda(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&)#1}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > >(milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long>()::{lambda(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&)#1}, std::function<bool (milvus::SkipIndex const&, fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, int)>, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> >) (this=this@entry=0x7fffeb505f10, func=..., skip_func=..., res=..., 
--Type <RET> for more, q to quit, c to continue without paging--
    values=std::unordered_set with 1 element = {...}, values=std::unordered_set with 1 element = {...})
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/Expr.h:265
#12 0x00007ffff47d98be in milvus::exec::SegmentExpr::ProcessDataChunksForSingleChunk<milvus::Json, milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long>()::{lambda(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&)#1}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > >(milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long>()::{lambda(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&)#1}, std::function<bool (milvus::SkipIndex const&, fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, int)>, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> >) (this=this@entry=0x7fffeb505f10, func=..., skip_func=..., res=..., 
    values=std::unordered_set with 0 elements, 
    values=std::unordered_set with 4195797946214473575 elements<error reading variable: Cannot access memory at address 0xffffffff00000000>)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/Expr.h:283
#13 0x00007ffff47d8ecf in milvus::exec::SegmentExpr::ProcessDataChunks<milvus::Json, milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long>()::{lambda(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&)#1}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > >(milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long>()::{lambda(milvus::Json const*, int, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> > const&)#1}, std::function<bool (milvus::SkipIndex const&, fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, int)>, milvus::bitset::BitsetView<milvus::bitset::detail::VectorizedElementWiseBitsetPolicy<unsigned long, milvus::bitset::detail::VectorizedDynamic>, true>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unordered_set<long, std::hash<long>, std::equal_to<long>, std::allocator<long> >) (this=this@entry=0x7fffeb505f10, func=..., skip_func=..., res=..., 
    values=std::unordered_set with 1 element = {...}, values=std::unordered_set with 1 element = {...})
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/Expr.h:399
#14 0x00007ffff47ccbe7 in milvus::exec::PhyJsonContainsFilterExpr::ExecJsonContains<long> (this=0x7fffeb505f10)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/JsonContainsExpr.cpp:260
#15 0x00007ffff47bd5b7 in milvus::exec::PhyJsonContainsFilterExpr::EvalJsonContainsForDataSegment (this=0x7fffedd7e004)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/JsonContainsExpr.cpp:84
#16 0x00007ffff47bcd72 in milvus::exec::PhyJsonContainsFilterExpr::Eval (this=0x7fffedd7e004, context=..., result=std::shared_ptr<milvus::BaseVector> (empty) = {...})
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/JsonContainsExpr.cpp:35
#17 0x00007ffff47b6c07 in milvus::exec::ExprSet::Eval (this=0x7fffeb5e0620, begin=<optimized out>, end=<optimized out>, initialize=<optimized out>, context=..., 
    results=std::vector of length 1, capacity 1 = {...}) at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/expression/Expr.cpp:42
#18 0x00007ffff4877241 in milvus::exec::PhyFilterBitsNode::GetOutput (this=0x7fffeac208a0)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/operator/FilterBitsNode.cpp:72
#19 0x00007ffff4630693 in milvus::exec::Driver::RunInternal (this=0x7fffeb164fb0, self=std::shared_ptr<milvus::exec::Driver> (use count 2, weak count 1) = {...}, 
    blocking_state=std::shared_ptr<milvus::exec::BlockingState> (empty) = {...}, result=std::shared_ptr<milvus::RowVector> (empty) = {...})
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/Driver.cpp:233
#20 0x00007ffff4634803 in milvus::exec::Driver::Next (this=0x0, blocking_state=<error reading variable: Cannot access memory at address 0xa>)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/Driver.cpp:162
--Type <RET> for more, q to quit, c to continue without paging--
#21 0x00007ffff4638d12 in milvus::exec::Task::Next (this=0x7fffeb504b00, future=0x0) at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/exec/Task.cpp:199
#22 0x00007ffff44a8246 in milvus::query::ExecPlanNodeVisitor::ExecuteTask (plan=..., query_context=std::shared_ptr<milvus::exec::QueryContext> (empty) = {...})
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/query/ExecPlanNodeVisitor.cpp:90
#23 0x00007ffff44ad5c5 in milvus::query::ExecPlanNodeVisitor::VectorVisitorImpl<milvus::BinaryVector> (this=0x7fffffffd6f0, node=...)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/query/ExecPlanNodeVisitor.cpp:137
#24 0x00007ffff4594cf6 in milvus::query::ExecPlanNodeVisitor::get_moved_result (this=0x7fffffffd6f0, node=...)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/query/ExecPlanNodeVisitor.h:63
#25 milvus::segcore::SegmentInternalInterface::Search (this=0x7fffea800700, plan=0x7fffeb21e360, placeholder_group=<optimized out>, timestamp=<optimized out>)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/src/segcore/SegmentInterface.cpp:88
#26 0x00005555559aced8 in ExprMaterializedViewTest::ExecutePlan (this=this@entry=0x7fffeb4d0110, plan=std::unique_ptr<milvus::query::Plan> = {...})
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/unittest/test_expr_materialized_view.cpp:166
#27 0x000055555598857f in ExprMaterializedViewTest_TestMvNotInvolvedJsonExpr_Test::TestBody (this=0x7fffeb4d0110)
    at /home/ubuntu/zilliz/milvus/milvus/internal/core/unittest/test_expr_materialized_view.cpp:371
#28 0x0000555555cc1319 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#29 0x0000555555ca78a8 in testing::Test::Run() ()
#30 0x0000555555ca88a0 in testing::TestInfo::Run() ()
#31 0x0000555555ca9365 in testing::TestSuite::Run() ()
#32 0x0000555555cb92dd in testing::internal::UnitTestImpl::RunAllTests() ()
#33 0x0000555555cc1f69 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#34 0x0000555555cb8e8a in testing::UnitTest::Run() ()
#35 0x00005555556a7c3a in RUN_ALL_TESTS () at /home/ubuntu/.conan/data/gtest/1.13.0/_/_/package/af5da70e9c6633f132bbd93d8fa9fb1384b7a7b6/include/gtest/gtest.h:2314
#36 main (argc=1, argv=0x7fffffffe098) at /home/ubuntu/zilliz/milvus/milvus/internal/core/unittest/init_gtest.cpp:31

Anything else?

No response

[yanliang567]

/assign @Presburger
/unassign

[xiaofan-luan]

I guess this is either related to local storage or null bitset

@sunby @smellthemoon

[sunby]

I guess this is either related to local storage or null bitset

@sunby @smellthemoon

local storage is disbaled by default.