Long-running Spark rewrite Files Action may lead to OutOfMemoryError

[Zhanxiao-Ma]

In my production environment, I have observed that long-running Spark rewrite Files Action can lead to OutOfMemoryError. Analyze the Java dump, I noticed a large number of ChildAllocator objects that are only referenced by the RootAllocator. Upon reviewing the code, I discovered that the ChildAllocator allocated at this point is indeed not being released. Is this correct?

iceberg/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java

Line 59 in cbb8530

this.rootAllocator =

[Zhanxiao-Ma]

I think the close method of VectorizedArrowReader should include the logic to release rootAlloc.