FAH v8.4.3 system tray program crash on Windows

[Hou5e]

FAH v8.4.3 system tray program crash on Windows that is probably related to #269. I had a Firefox web browser tab open viewing 2-10 remotes for about a day when this happened.

There was an error window open for debugging, this time (I picked VS2022, and I should have picked VS2015 for debugging).

The top few long messages in the call stack image are these lines of text:

FAHClient.exe!std::_Tree<std::_Tset_traits<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>,std::less<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>>,std::allocator<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>>,0>>::_Find_lower_bound<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>>(const cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>> & _Keyval) Line 1590
	at C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\include\xtree(1590)

FAHClient.exe!std::_Tree<std::_Tset_traits<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>,std::less<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>>,std::allocator<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>>,0>>::_Emplace<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>> const &>(const cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>> & <_Vals_0>) Line 1003
	at C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\include\xtree(1003)

FAHClient.exe!std::_Tree<std::_Tset_traits<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>,std::less<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>>,std::allocator<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>>>,0>>::insert<0,0>(const cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>> & _Val) Line 1222
	at C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\include\xtree(1222)

FAHClient.exe!cb::LifetimeManager::addLTO<cb::LifetimeObject>(const cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>>> & obj) Line 56
	at C:\build\fah\windows-10-64bit-client-bastet-debug\cbang\src\cbang\util\LifetimeManager.h(56)

FAHClient.exe!cb::WS::Websocket::readBody() Line 276
	at C:\build\fah\windows-10-64bit-client-bastet-debug\cbang\src\cbang\ws\Websocket.cpp(276)

FAHClient.exe!cb::WS::Websocket::readHeader::__l2::void <lambda>(bool)::__l2::<lambda>(bool success) Line 214
	at C:\build\fah\windows-10-64bit-client-bastet-debug\cbang\src\cbang\ws\Websocket.cpp(214)

[External Code]

[jcoffland]

Damn it. This is a similar crash to what we were seeing before on Windows in #235.

Note, there is no separate systray program. The systray icon is created by FAHClient.exe.

[jcoffland]

On further investigation this appears to be different from the #235 crash. I suspect a memory corruption. Perhaps introduced with v8.4.2.

[jcoffland]

If you can capture a log file and/or another stack trace that would be great.

[Hou5e]

It hasn't reproduced again yet (in 4 days). I was running the Debug v8.4.3 when it happened, and I'm running the Release v8.4.4 now. I just switched to the v8.4.4 Debug version and started running some 0xfe WUs periodically, to see if the issue will happen faster than 7-10 days.

This was the log for the day before: log-20240827-000003.txt
I think it crashed at the end of this log: log-20240827-201354.txt
There aren't many clues to work from in it. It looks like the PC sat idle overnight, and was just starting Folding again, then FAH died. I only want this PC to fold from about 10:45am to 2:00pm on solar electricity. For the automated FAH start and pause, I have a Bash script on Windows that runs an older lufah single file with: python /c/Projects/FAHTimeMonWin/lufah.py ./RG2 unpause

[Hou5e]

OK, it happened again today. The stack trace seems exactly the same (in VS2015 this time). The longer lines at the top of of the call stack are:

FAHClient.exe!std::_Tree<std::_Tset_traits<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > >,std::less<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > >,std::allocator<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > >,0> >::_Find_lower_bound<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > >(const cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > & _Keyval) Line 1590	C++

FAHClient.exe!std::_Tree<std::_Tset_traits<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > >,std::less<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > >,std::allocator<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > >,0> >::_Emplace<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > const &>(const cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > & <_Vals_0>) Line 1003	C++

FAHClient.exe!std::_Tree<std::_Tset_traits<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > >,std::less<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > >,std::allocator<cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > >,0> >::insert<0,0>(const cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > & _Val) Line 1222	C++

FAHClient.exe!cb::LifetimeManager::addLTO<cb::LifetimeObject>(const cb::SmartPointer<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject>,cb::RefCounterImpl<cb::LifetimeObject,cb::DeallocNew<cb::LifetimeObject> > > & obj) Line 56	C++

Log messages: log-20240902-202112-Partial_Log.txt

There was one local FireFox browser tab viewing remotes for ~2 days with 50-100MB log files per day.

This seems very suspect, like it has to do with the extra WebSocket connection (not being disposed cleanly?):
FAH died about 1-2 minutes after the 'Finish' command was issued with (old) Lufah in a Bash script.
20:19:36:D3:msg: {"cmd": "finish"}
20:19:36:I1:Machine state finish

NOTE: From the previous crash log, FAH died almost exactly 1 minute after (old) Lufah issued 'Start' from a script.
17:47:58:D3:msg: {"cmd": "unpause"}
17:47:58:I1:Machine state fold

[Hou5e]

The same crash (the call stack and image of the error are the same as above) happened again twice today.
Mid-day crash when FAH and the PC was busy: log-20240904-084353-Partial.txt
Late night crash after finishing the WU that crashed earlier, where FAH and the PC were somewhat idle: log-20240904-095127.txt

The automated pause / resume commands were not issued near when either of these crashes happened.

[jcoffland]

Is this still occurring?

[Hou5e]

This specific crash with an error message hasn't happened in the last 3 weeks (I wasn't running FAH on Windows as much). But, I'm pretty sure the system tray program disappeared (died with no error or log message) and FAH was still running in the background (needing to be stopped with the task manager) for the latest FAH v8.4.5, about a day after I updated from FAH v8.4.4 to the latest (on about 9/18).