Quix Streams Release 3.20.0

Quix Streams 3.20.0 delivers transaction control, enhanced configuration management, and PostgreSQL conflict resolution to help engineering teams build more reliable data processing systems.

Enhanced transaction control for mission-critical R&D workloads

Transaction isolation: Atomic processing prevents partial data corruption during complex test runs

R&D environments generate massive amounts of interdependent data from test rigs, simulations, and prototype builds. When processing telemetry data from a rocket engine test or battery performance evaluation, partial failures can corrupt entire datasets. The new transactional parameter in Application.get_producer() enables Kafka's transaction API, ensuring all related messages are processed atomically or not at all.

from quixstreams import Application

app = Application(broker_address="localhost:9092")

# Enable transactions for atomic message production
producer = app.get_producer(transactional=True)

# Process critical test data with guaranteed consistency
with producer.transaction():
    producer.produce("sensor-data", key="test-123", value=sensor_readings)
    producer.produce("analysis-results", key="test-123", value=calculations)
    # Both messages committed together or rolled back on failure

‍


This prevents scenarios where sensor data arrives but analysis results are lost, eliminating the manual effort required to identify and reprocess incomplete test runs.

Workspace isolation: Automatic transaction ID prefixing prevents cross-project data mixing

Multi-project R&D organizations often run parallel experiments on shared infrastructure. The automatic prefixing of transactional.id by workspace ID when connecting to Quix brokers ensures transaction isolation between different teams and projects. A battery testing team's transactions won't interfere with an HVAC optimization project running on the same cluster, reducing the risk of data contamination between experiments.

Flexible configuration management for dynamic R&D environments

Binary and JSON field support: Handle diverse configuration data types from engineering tools

R&D teams work with configuration data from multiple sources: MATLAB models, CAD parameters, test specifications, and calibration files. The enhanced lookup.quix_configuration_service now supports both binary and JSON fields, allowing you to store and retrieve diverse configuration types in a single system.

# Store binary calibration data alongside JSON parameters
config_service = lookup.quix_configuration_service(
    workspace_id="battery-testing",
    project_id="lithium-ion-optimization"
)

# Retrieve mixed configuration data
test_config = config_service.get("test-parameters")
calibration_binary = config_service.get("sensor-calibration")  # Binary data
analysis_settings = config_service.get("analysis-config")      # JSON data

‍This eliminates the need to maintain separate storage systems for different data types, reducing infrastructure complexity and improving data traceability across the entire R&D workflow.

Version handling improvements: Robust configuration versioning for experimental iterations

R&D experiments often span weeks or months with frequent parameter adjustments. The improved handling of valid_from=None in configuration version selection provides more predictable behavior when accessing historical configurations. This ensures you can reliably reproduce any experimental iteration, even when configuration timestamps are incomplete or missing.

Database conflict resolution for collaborative R&D data

PostgreSQL upsert operations: Handle concurrent data writes from distributed test systems

Modern R&D involves distributed teams running parallel tests on similar systems. Multiple test rigs might simultaneously write performance data for the same component or test condition. The new ON CONFLICT DO UPDATE support in PostgreSQLSink automatically resolves these conflicts by updating existing records instead of failing.

from quixstreams.sinks.community.postgresql import PostgreSQLSink

# Configure automatic conflict resolution
postgresql_sink = PostgreSQLSink(
    host="test-results-db",
    database="component_testing",
    table="performance_metrics",
    # Automatically update on primary key conflicts
    on_conflict="DO UPDATE"
)

# Multiple test rigs can write to the same component_id
# Latest results automatically update previous entries
sdf.sink(postgresql_sink)

‍This prevents the manual intervention typically required when multiple teams test the same components, allowing faster iteration cycles and reducing data pipeline maintenance overhead.

Upgrading to Quix Streams 3.20.0

Update your environment using pip:

pip install --upgrade quixstreams

The transaction features require no code changes for existing applications but provide additional reliability options when processing mission-critical R&D data. Configuration service improvements are backward compatible, while the PostgreSQL conflict resolution requires explicit configuration to enable.

Next steps

These reliability and configuration improvements help engineering teams transition from desktop-based R&D workflows to centralized data platforms that support collaborative development and automated processing. R&D data infrastructure needs robust transaction handling, flexible configuration management, and conflict resolution to support the complex, iterative nature of product development.

Explore the Quix Streams documentation for implementation details, or review the complete release notes for additional technical information.

‍