Change Data Capture with Neon serverless Postgres, Debezium, and Redis

This repository is a companion to a blogpost on the Neon Blog. It provides a template for getting started with Neon and Debezium to perform Change Data Capture.

Using Change Data Capture enables you to stream database changes to messaging infrastructure such as Kafka, Redis, and Pub/Sub systems without suffering from data inconsistency caused by dual writes. Downstream systems can consume these messages to enable Event-Driven Architectures.

Getting Started

1. Obtain a Neon Postgres database.
2. Obtain a Redis instance from Upstash.
3. Define environment variables.
4. Start Debezium Server.
5. Confirm that changes are streamed to Redis.

Configure Neon Postgres

1. Sign up to console.neon.tech and create a project to obtain a serverless Postgres database.
2. Enable Logical Replication for your Neon project.
3. Create a table and insert data using Neon's SQL Editor:

   CREATE TABLE playing_with_neon(id SERIAL PRIMARY KEY, name TEXT NOT NULL, value REAL);
   
   INSERT INTO playing_with_neon (name, value)
   VALUES 
   ('Mario', random()),
   ('Peach', random()),
   ('Bowser', random()),
   ('Luigi', random()),
   ('Yoshi', random());

Configure Upstash Redis

1. Sign up and create a Redis database on console.upstash.com with the following settings:
   • Name: neon-debezium
   • Type: Regional
   • Region: Select the region closest to your Neon Postgres database.
   • TLS (SSL) Enabled: Yes
   • Eviction: Yes

Set Environment Variables

1. Copy the .env.example to a file named .env
2. Replace the Postgres connection parameters in .env with your own values from Neon
3. Replace the Redis connection parameters with your own values from Upstash.

Start Debezium

Start a Debezium Server container, passing the .env file and application.properties to it:

docker run --net neon-debezium-redis \
--rm \
--name debezium-server \
--env-file=.env \
-v $PWD/debezium:/debezium/conf \
debezium/server:2.5.1.Final

A few moments after Debezium starts, you should see that it prints logs that confirms it has performed an initial snapshot of the playing_with_neon table, and is now listening for WAL changes.

Confirm Changes are Streamed to Redis

Visit the Data Browser for your Redis instance in Upstash and you should see that a debezium.public.playing_with_neon Redis stream has been created, and contains records corresponding to INSERT events from your playing_with_neon table.

Perofrming more INSERT or UPDATE operations will result in new messages appearing in the Redis stream.

Consuming the Change Stream

A sample application is included in the stream-consumer/ folder. It uses the Redis Node.js client to consume the stream of database changes in the debezium.public.playing_with_neon Redis key, and create a sum of "scores" for each player in Redis.

For example, performing the following two INSERT operations would result in a sum:mario key being created in Redis with a value of 2.1:

INSERT INTO playing_with_neon (name, value) VALUES ('Mario', '1.5')
INSERT INTO playing_with_neon (name, value) VALUES ('Mario', '0.6')

To run the Node.js application:

1. Create a copy of the .env.example named .env in thw stream-consumer/ folder.
2. Replace the sample values with your Upstash Redis connection parameters. Note that rediss:// is the correct protocol to use if SSL is enabled for your Upstash Redis instance.
3. Run npm install to install dependencies.
4. Run npm run to start the application.

Once the application starts it will continuously update Redis keys as INSERT events are written to the stream key by Debezium.