A collection of Kafka Connect Single Message Transforms (SMTs)

These SMTs are designed to be used with the Kafka Connect framework. The SMTs create record headers. The advantage of using headers is that they reduce the memory and CPU cycles required to change the payload. See for example the Kafka Connect TimestampConverter. Furthermore, they support Stream-Reactor S3 sink partitioner, for scenarios like:

• Partitioning by system clock (e.g. using the system clock as a partition key with a yyyy-MM-dd-HH format)
• Partitioning by a rolling window (e.g. every 15 minutes, or one hour)
• Partitioning by a custom timestamp (e.g. a timestamp field in the payload, record Key or Value)
• Partitioning by a custom timestamp with a rolling window (e.g. a timestamp field in the payload, every 15 minutes, or one hour)

SMTs

• InsertWallclock - Inserts the system clock as a message header.
• InsertRollingWallclock - Inserts the system clock as a message header based on a rolling window boundary.
• InsertRollingRecordTimestampHeaders - Inserts date, year, month, day, hour, minute, and second headers using the record timestamp and a rolling time window configuration.
• InsertRollingWallclockHeaders - Inserts date, year, month, day, hour, minute, and second headers using the system timestamp and a rolling time window configuration.
• InsertRecordTimestampHeaders - Inserts date, year, month, day, hour, minute, and second headers using the record timestamp.
• InsertFieldTimestampHeaders - Inserts date, year, month, day, hour, minute, and second headers using a field in the payload, record Key or Value.
• InsertRollingFieldTimestampHeaders - Inserts date, year, month, day, hour, minute, and second headers using a field in the payload, record Key or Value and a rolling window boundary.
• InsertWallclockHeaders - Inserts date, year, month, day, hour, minute, and second headers using the system clock.
• TimestampConverter - Converts a timestamp field in the payload, record Key or Value to a different format, and optionally applies a rolling window boundary. An adapted version of the one packed in the Kafka Connect framework.
• InsertWallclockDateTimePart - Inserts the system clock year, month, day, minute, or seconds as a message header, with a value of type STRING.

Installation

The build jar can be found in the releases. To install the jar, copy it to the plugin.path directory of your Kafka Connect worker.

Build

To build the project run:

mvn clean package

Checkstyle

To check the code style run:

mvn checkstyle:check

Formatter

To format the code run:

  mvn com.coveo:fmt-maven-plugin:format

License Headers

To add license header, run:

mvn license:format

Dependency Check

To run the dependency check:

1. Getting NVD API Key:

   • Visit the National Vulnerability Database (NVD) Website, sign up or log in to obtain your API key.

2. Setting NVD API Key in Environment Variable:

   • Mac (zsh): Add export NVD_API_KEY=your_api_key_here to ~/.zshrc.
   • Windows: Use setx NVD_API_KEY "your_api_key_here" in Command Prompt (Admin).
   • Linux (bash): Add export NVD_API_KEY=your_api_key_here to ~/.bashrc.

3. Running mvn verify for Dependency Check Plugin:

   • Run mvn verify to execute it. This will put a dependency-check-report.html in the target directory which will give you details of the scan result.