Enrich data with a Redis lookup
A simple three-step pipeline that you can use as a basis for any project. It streams data from a small CSV to a Kafka topic. From there, it’s consumed by a basic aggregation process which writes the output to a second topic. The output is consumed by a basic logging service.
![A simple real-time pipeline diagram](https://cdn.prod.website-files.com/64a7eed956ba9b9a3c62401d/6654e9bcccc8a8e5490a9cef_template_image_enrich.jpg.png)
Main project components
CSV data source
A source component that streams a basic time-series dataset (containing mock sensor data) to a Kafka topic.
Data enrichment
A transformation service that performs a real-time lookup on Redis data for each processed message and appends the extra data to the message.
Console logger
A simple consumer service that reads from the “enriched-data” topic and logs the output to the console.
Lookup data ingestion
A one-time job that ingests a CSV file containing mock sensnor coordinates into a Redis database.
Technologies used
Using this template
This is a simple base template that you can easily adapt for any use case that requires data enrichment.