Excessive memory usage when using KotlinModule with Kotlin data class

[t9t]

Search before asking

• I searched in the issues and found nothing similar.
• I searched in the issues of databind and other modules used and found nothing similar.
• I have confirmed that the problem only occurs when using Kotlin.

Describe the bug

When using KotlinModule with an ObjectMapper to unmarshall a Kotlin data class, it uses a lot more memory than when not using KotlinModule or when not using a Kotlin data class (e.g. when using a Java record).

To Reproduce

Please see: https://github.com/t9t/jackson-kotlin-memory

Expected behavior

Lower memory overhead when reading Kotlin data classes. Especially when using Jackson annotation.

Versions

Kotlin: 1.9.22
Jackson-module-kotlin: 2.16.1
Jackson-databind: 2.16.1

Additional context

We found this out in a backend message processing app, which was crashing due to high CPU usage spend in the garbage collector. Upon inspection, we were parsing JSON payloads in the 10KiB-20KiB range (hundreds per second), and generating dozens of MiBs of garbage per second.

When playing with a profiler locally, I noticed a lot of allocations in Jackson Kotlin code. I was able to distill a small test project that shows the issue: https://github.com/t9t/jackson-kotlin-memory

There is approximately 2x the memory overhead when using KotlinModule with a Kotlin data class vs not using KotlinModule (with one of our specific 20 KiB payloads, the overhead with KotlinModule vs without was 4x).

The most surprising thing for me is that adding the Jackson annotations (@JsonCreator and @JsonProperty) did not help with the issue. When I use a profiler, I see many allocations in Kotlin reflection functions (but I have to confess I'm yet a novice in using profilers). I would expect that when using the annotations, no reflection would be needed.

[k163377]

First, most of the memory consumption comes from kotlin-reflect.
Almost all of the content held within kotlin-reflect is SoftReference and should be freed when it is no longer needed.
If there is a bug in the kotlin-module implementation that is consuming more memory, please point it out to us.

The most surprising thing for me is that adding the Jackson annotations (@JsonCreator and @JsonProperty) did not help with the issue. When I use a profiler, I see many allocations in Kotlin reflection functions (but I have to confess I'm yet a novice in using profilers). I would expect that when using the annotations, no reflection would be needed.

The analysis required for Kotlin cannot be covered by annotations alone.
If there is any analysis that is not needed internally, we welcome your PR.

[t9t]

Thank you very much for looking into the issue.

I am a user of kotlin-module and as I was using it, I noticed the issue of higher memory overhead when using KotlinModule than when not using it. I currently am not knowledgeable about the internals of jackson-module-kotlin or its dependency graph, so I decided to report the issue here, hoping that the report would be useful to you. I don't know if it's a bug in jackson-module-kotlin, or in any of its dependencies, but it is an issue that I am experiencing while using KotlinModule.

Please note that the issue is not about memory usage, but about allocations. The problem is precisely that a lot of memory is allocated (and then freed), which is causing memory overhead, which causes the garbage collector to perform work.

It may well be that there is no bug at all, but perhaps there are allocations that are performed each time during deserialization that could be (optionally) cached to lower the allocation overhead (with the cost of more persistent memory usage). Especially when using reflection on types, it seems wasteful to perform the reflection each time. Perhaps there is a technique I can use already to avoid the reflection and the associated memory overhead.

The analysis required for Kotlin cannot be covered by annotations alone.

The reproducible case that I posted shows that using a Kotlin data class with an ObjectMapper without KotlinModule but with Jackson annotations (@JsonCreator and @JsonProperty) works without issues, and with a lower allocation footprint than when using KotlinModule. This leads me to believe that it should be possible to somehow detect this case, and not perform the expensive reflection when the annotations are present.

Please let me know if you require any more information.