Quix Streams Release 3.18.0

Quix Streams v3.18.0: Introducing PostgreSQL Lookup Joins for Real-time Data Enrichment

We're excited to announce the release of Quix Streams v3.18.0, which introduces a powerful new feature for enriching streaming data with PostgreSQL lookups. This release continues our program of making it easier and faster to join time-series data with other real-time data or slow moving and static data.

‍PostgreSQL Lookup Joins

The highlight of this release is the new `PostgresLookup`, which enables enrichment of streaming data with information from PostgreSQL databases. This feature is particularly valuable for scenarios where you need to enhance streaming data with reference data or additional context stored in a relational database.

Key Benefits

1. Efficient Data Enrichment: Enrich streaming data with reference data from PostgreSQL in real-time

2. Built-in Caching: Implements an LRU (Least Recently Used) cache with configurable TTL to minimize database load

3. Persistent Connections: Maintains efficient database connections for optimal performance

4. Simple Integration: Easily add database-backed lookups to your streaming pipelines with minimal code

How It Works

Here's a quick example of how to use the new PostgreSQL Lookup feature:

from quixstreams import Application 
from quixstreams.dataframe.joins.lookups.postgresql import PostgresLookup 

app = Application(...) 
sdf = app.dataframe(...) 

# Initialize PostgresLookup with your database credentials
lookup = PostgresLookup(
    host="<host>",
    port=5432,
    dbname="<db>",
    user="<user>",
    password="<password>",
    cache_size=1000,
)

# Define the fields to join from the database
fields = {
    "joined": lookup.field(
        table="my_table",
        columns=["table_column1", "table_column2"],
        on="table_column1"
    ),
}

# Perform the lookup join
sdf = sdf.join_lookup(lookup, fields, on="my_record_field")

app.run()

‍‍

Use Cases

- Real-time Customer Data Enrichment: Enhance streaming events with customer profiles stored in PostgreSQL

- Reference Data Lookups: Join streaming data with reference data like product catalogs or pricing information

- Contextual Enrichment: Add additional context to events based on historical data

‍Bug Fixes

- Fixed a typo in the Producer class

‍Getting Started

To start using the new PostgreSQL Lookup feature, upgrade to Quix Streams v3.18.0:

pip install --upgrade quixstreams

‍

For more detailed information, check out the PostgresLookup API documentation.

We're excited to see how you'll use this new capability to enhance your real-time data processing pipelines. If you have any questions or feedback, feel free to reach out on our GitHub repository. (And please leave a⭐)

‍