Increased amount of ANRs after disabling concurrent GC #9365
Comments
TimBurik
added
Area: App Runtime
|
@TimBurik I'm afraid we can't do anything for you here, this appears to be an issue split between MonoVM and the Sentry Native SDK. @lambdageek do you think someone on your side could take a look to see if anything can be done here? There's an issue being worked on related to Sentry, #9055, perhaps this here issue is also caused by the same problems? @supervacuus, sorry for tagging you out of the blue, but would you be able to look into the Sentry side of things here? I realize we have precious little information, but perhaps someone will be able to spot something in their respective areas and help out. |
grendello
added
Area: Mono Runtime
|
@grendello thank you for the response. Regarding #9055 - we are aware of this issue, and we have a separate native crash group, which is most likely related to this issue. But the ANRs doesn't seem to be related to this issue and to Sentry, because we are seeing exactly the same picture in the GooglePlay Console - increased amount of ANRs, all containing |
|
The GC ANRs might be related to something inside the GC bridge, hopefully @lambdageek will be able to help out here. However, this ANR looks familiar: You are most likely not using marshal methods, since they weren't enabled by default in NET8, but we've fixed an issue recently which was related to them and one of the repros had the above stack trace as well. The problem was related to Java ("native") threads being incorrectly attached to the MonoVM runtime and putting the GC in a bad state. You can find more info about it in this and the following comments. @filipnavara, does this look familiar? Perhaps something in your app causes a similar corruption of the GC state? @TimBurik if you're able to reproduce the ANRs locally, would you be able to test with NET9 rc2? |
|
@grendello thank you for the pointers! We are planning to test the app on .NET 9 in the nearest future, we would make sure to use RC2 for those tests (when RC2 would be available). Also, we have added debug symbols for libmonosgen-2.0 into GooglePlay Console and now we have more detailed logs. For example, this is the reported main thread stack trace from one of the ANRs: |
|
@TimBurik this more and more looks like the issue fixed by @filipnavara, he declared he'd take a look next week - it would be great if you were able to test with NET9 RC2 before then :) |
|
@TimBurik if RC2 isn't available by next week, do try with RC1 - it's to see if the issues you're seeing still even exist in NET9 |
|
I've been following the thread since you tagged me. It seems this isn't sentry-related. I've informed sentry peeps, just in case. |
|
@supervacuus thanks, I tagged you just in case - this is all low-level stuff, it might be affecting Sentry as well in some ways. |
|
/cc @vitek-karas |
|
@grendello sorry for late response |
|
I don't know how the ANR system works, I would imagine they happen if the app is non responsive for a while. If this is the case, I'm not sure whether this issue can actually be considered to be a bug. We seem to have a problem with the concurrent GC in the runtime and this issue needs to be investigated somehow. Disabling concurrent GC will just lead to longer GC pause times which seem to trigger this ANR. Is a GC pause time of 0.5 - 1s supposed to trigger this ANR ? If this is true, it seems to me that the best way to move forward would be to share the application so we can see if the serial GC takes longer than expected for whatever reason and mainly to try some debug flags to see if we detect some problems with the concurrent collector. |
|
@BrzVlad according to the Android documentation, one of the reasons for ANR is when BroadcastReceiver has failed to process an event within 5 seconds - this seems to be our case, as there's an additional description in the GooglePlay Console like "Broadcast of Intent { act=android.intent.action.SCREEN_ON }" By the way, let me share a full report from the GooglePlay Console, which contains stack traces from all the threads: anr_all_stacktraces.txt I could also try to gather additional information regarding the GC performance, but in general we didn't notice any significant performance degradation after disabling concurrent GC |
|
@grendello in our case most of the ANRs are related to the But this is not the only source for ANRs for us, as we also have reports:
All of those reports have very similar main thread stack traces, related to GC. It's even seems that those broadcast events, service calls and input events are not really causing the issue, and the main thread is already blocked by the time these events occur. |
|
@grendello @BrzVlad is there anything else we could do to help you shed the light on this issue? |
|
Seing that this issue happens after disabling concurrent GC I was biased into assuming that the GCs simply take longer to run and this is causing issues. After a second look at the stacktraces, it indeed looks like the suspend machinery is hanging. It makes no sense to me why this issue would only happen with serial GC though. We have an env var |
|
@BrzVlad could you also give a hint to where the threads dump could be found? In logcat we see the message: We are using the following values as Android environment variables: and also we are using the |
|
I would have expected that the suspend state of each thread to be printed but it looks like that logging path doesn't go through the android log but rather to the console. It would be useful for us to try to reproduce this failure but given the abort limit is not excessive and it only happens while in background, it makes me wonder whether it just the os scheduling the app less frequently and the suspend process just taking a bit longer than usual, because threads reach GC safepoint location slower. |
|
This issue doesn't seem to affect every GC in the background, in the log cat we also see a couple of GC-related logs: I'm also gonna try to reproduce this issue with bigger values for |
|
@BrzVlad Just managed to reproduce the issue with limit value of 1s: The reproduction steps are the same - put application to the background and wait, although this time it required also several tries (foregrounding/bacgrounding app back) and considerable amount of waiting (30m+ after each backgrounding) |
|
Would it be possible to provide the sample app with some reproduction steps ? |
|
Let me try to reproduce the issue on the sample app, and also discuss with the legal team possibility to share our production code. Meanwhile, I have found a similar issue with a deadlock in GC stopping the world mono/mono#21582, which ended up being fixed by changing the value of the |
|
That is a good observation. The default suspend mode should by |
|
@BrzVlad We have just tested The bad news is that instead, we now see a brand new ANR groups, related to I'm also attaching full stacktraces just in case: |
Android framework version
net8.0-android
Affected platform version
.NET 8.0.303
Description
After switching from Xamarin.Android to .Net8 we used to get a lot of native crashes in monosgen, for example 222003f52fa3496385d14a89c778a6e4-symbolicated.txt
After long investigation (and a hint from the issue dotnet/runtime#100311) It turns out that concurrent SGen is actually enabled by default in .net-android
android/src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.DefaultProperties.targets
Line 9 in df9111d
so we explicitly disable it - and now the amount of native crashes in monosgen is minimal, but instead we are getting a lot of ANR reports in Sentry and GooglePlay Console.
ANRs seems to be reported using Android's ApplicationExitInfo mechanism, according to stacktraces main thread seems to be blocked by awaiting native mutex somewhere in the monosgen (example: anr_stacktrace.txt)
Additional information, which might be relevant:
Intent { act=android.intent.action.SCREEN_OFF }orIntent { act=android.intent.action.SCREEN_ON };Steps to Reproduce
Unfortunately, we don't have exact steps to reproduce.
The only thing that is sure that it is happening when targeting
.net-android34.0(version for Xamarin.Android doesn't have this issue) and issue started happening after adding the following to the csproj:<AndroidEnableSGenConcurrent>false</AndroidEnableSGenConcurrent>Did you find any workaround?
No workaround found yet
Relevant log output
No response
The text was updated successfully, but these errors were encountered: