You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
FROM confluentinc/cp-kafka-connect-base:7.4.3
ENV CONNECT_PLUGIN_PATH "/usr/share/java"RUN confluent-hub install --no-prompt confluentinc/kafka-connect-s3:10.5.7
RUN cp -r /usr/share/confluent-hub-components/confluentinc-kafka-connect-s3 /usr/share/java
We deployed it on Kubernetes as a deployment with 8 replicas. We created a connector that consumes 3 topics with 400 partitions in total. It is reading the data from those topics and writing the data into AWS S3 as hourly partitioned parquet files. We were getting OOM kills frequently, so we started to the changes below:
Reduce s3.part.size to the minimum value of 5 mib
Enable elastic buffer
Reduced rotate.interval.ms from one hour to only 20 minutes.
We didn't even need to touch flush.size as the partition files were getting very small already.
Reducing the rotation interval by 5x didn't even reduce the memory usage of our deployment. That made us very suspicious. So we enabled JMX metrics and started to play around with JVM settings:
Hi,
We have a Docker image like below:
We deployed it on Kubernetes as a deployment with 8 replicas. We created a connector that consumes 3 topics with 400 partitions in total. It is reading the data from those topics and writing the data into AWS S3 as hourly partitioned parquet files. We were getting OOM kills frequently, so we started to the changes below:
s3.part.size
to the minimum value of 5 mibrotate.interval.ms
from one hour to only 20 minutes.flush.size
as the partition files were getting very small already.This is what the connector config looks like:
Reducing the rotation interval by 5x didn't even reduce the memory usage of our deployment. That made us very suspicious. So we enabled JMX metrics and started to play around with JVM settings:
Here is what the metrics look like:
We noticed that the direct memory buffer is the source of our always-increasing K8s deployment memory usage.
I will add more info if I get a chance to analyze some heap dumps.
Questions:
The text was updated successfully, but these errors were encountered: