Enrich data with a Redis lookup
A simple three-step pipeline that you can use as a basis for any project. It streams data from a small CSV to a Kafka topic. From there, it’s consumed by a basic aggregation process which writes the output to a second topic. The output is consumed by a basic logging service.
Main project components
CSV data source
A source component that streams a basic time-series dataset (containing mock sensor data) to a Kafka topic.
Data enrichment
A transformation service that performs a real-time lookup on Redis data for each processed message and appends the extra data to the message.
Console logger
A simple consumer service that reads from the “enriched-data” topic and logs the output to the console.
Lookup data ingestion
A one-time job that ingests a CSV file containing mock sensnor coordinates into a Redis database.
Technologies used
Using this template
This is a simple base template that you can easily adapt for any use case that requires data enrichment.
