Topic, message and key from columns

[walec51]

Problem

We wanted to use this JDBC Connector not to produce messages that expose our database schema but clean JSON messages that have been prepared in a database table after a transaction was commited.

We also wanted those JSON messages also indicate a destination topic and key on a message to message basis.

Thanks to this we can implement this pattern:

https://vimeo.com/111998645?embedded=true&source=video_title&owner=19072516

using this connector

Solution

Adding some configuration options with which we can indicate which table columns hold:

• the destination topic
• the messages value
• the messages key

Does this solution apply anywhere else?

• yes
• no

If yes, where?

Every system this whats to implement this pattern using this connector:

https://vimeo.com/111998645?embedded=true&source=video_title&owner=19072516

Test Strategy

Testing done:

• Unit tests
• Integration tests
• System tests
• Manual tests

Release Plan

We would like to merge this to master.

We are aware that this PR still needs some work and would be grateful for some early feedback and commends.

We develop this code with @Mistral and @Lukier1 and we are already using it on our test environments at ewejsciowki.pl

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 3 committers have signed the CLA.

❌ Lukier1
❌ Mistral
❌ Adam Walczak

---

Adam Walczak seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[OneCricketeer]

Ideally, you'd use a series of custom transformers to achieve this same result. Such as ValueToKey, and one that overrides the ConnectRecord partition, etc.