Elevate time series data analysis with InfluxDB & Python stream processing from Quix

How do Quix and InfluxDB work together?

Here’s how InfluxDB and Quix work together from an architectural perspective:

• InfluxDB acts as the time series database, which is responsible for ingesting time series data from sources (through Telegraf), feeding it to the stream processing component for real-time processing, and then receiving the transformed data post-processing to store and operationalize it.
• Quix is a Python-based stream processing engine that consumes time series data from InfluxDB (or from various sources via Telegraf and Quix connectors), transforms it in real time, publishes the output back to InfluxDB, and enables advanced real-time capabilities.
• InfluxDB is available as a fully managed solution - this greatly reduces the effort required to handle time series data. Quix is also a managed platform, removing the headache of operating a high-performance application back-end that processes time series data in-house.

Integrating Quix and InfluxDB requires minimum effort. This is made possible by ready-made Quix source and sink connectors that allow you Quix to read from and publish to InfluxDB:

• InfluxDB 2.0 source connector
• InfluxDB 3.0 source connector
• InfluxDB 3.0 sink connector

The stream processing logic is implemented using Quix Streams, an open-source technology that combines an Apache Kafka client with a Python stream processing library. You can think of Quix Streams as an all-in-one replacement for Flux and Kapacitor. It has the following key capabilities: 

• Librdkafka-compatible. Low-level Kafka producer and Kafka consumer classes to read from and write to Kafka topics.
• Easy to learn. Intuitive Streaming DataFrame API (similar to pandas DataFrame) for tabular data transformations.
• Windowing. Aggregations over hopping and tumbling windows
• Stateless processing. This includes grouping, filtering, projections, and dropping columns.
• Custom processing. The ability to transform data from one format to another and implement custom, user-defined processing functions.
• Stateful operations. Stateful processing uses RocksDB for state storage.
• Simple output routing. The ability to succinctly produce data to output Kafka topics using the to_topic() method.
• Flexible processing guarantees. At-least-once and exactly-once processing semantics.
• Versatile serialization. Support for various data serialization formats: bytes, string, integer, double, JSON, Protobuf, and Avro.
• Simplified deployment. Seamless integration with Quix Cloud, a fully managed platform offering a frictionless environment for deploying and managing your Python stream processing applications.

For additional information and detailed, step-by-step guidance on how these two technologies work together in practice, check out the Quix & InfluxDB quickstart guide.

What are the benefits of using Quix alongside InfluxDB?

Smooth development experience

Ensuring developer happiness and productivity are key considerations for the team behind InfluxDB. Quix shares this ethos and strives to offer a simple, straightforward experience to Python developers working with time series data:

• The Quix Python Streaming DataFrame API is intuitive and easy to use, especially if you’re familiar with pandas DataFrames
• There’s no server-side engine
• You can easily integrate the entire Python ecosystem into your Quix pipelines (scikit-learn, TensorFlow, PyTorch, etc)
• Quix offers comprehensive documentation that’s regularly updated, ready-to-run code samples and templates to help you get started in minutes, and various resources like webinars, tutorials, and videos to aid you in your Python stream processing journey

There’s a growing, active, and responsive Quix community to help with any issues you might encounter.

{{testimonial_Jay-Clifford}}

Flexible, comprehensive tooling

InfluxDB offers comprehensive tooling to help developers manage time series data more easily. Examples include the InfluxDB UI, CLI tools, prepackaged configurations (templates and stacks), and a single binary for straightforward deployment.

Similarly, Quix offers everything you need to seamlessly build, deploy, and manage industrial-strength applications that process time series data: 

• CI/CD support.  Can integrate with any Git provider (e.g., GitHub, Bitbucket, Azure DevOps) for seamless CI/CD processes.
• Environment control. Multiple projects and environments (linked to Git) for streamlined environment management.
• Team collaboration. Multi-user collaboration at project and environment levels through organization and permission management.
• Infrastructure management.  Infrastructure as code (IaC) using Quix YAML (similar to Helm charts) with automated synchronization.
• Observability and monitoring. Capabilities to monitor your data pipelines, such as real-time logs, metrics, data explorers, and waveform and table views.
• Security. Securely manage secrets and sensitive information.
• Dev tools. Online code editor, code templates, and connectors for various data sources and sinks, such as Redis, MQTT brokers, and Kafka Connect.
• Pipeline management. Functionality to scale resources, configure replication, and manage CPU and memory for your data pipelines.
• Support for various Kafka brokers. Quix-hosted Kafka, Confluent Cloud, Redpanda, Aiven, Upstash, and self-hosted Kafka.‍
• Local development. CLI tool to create, debug, and run your data pipeline locally, then deploy it to the cloud using only the command line.

{{testimonial_Carey-McLean}}

Reduced costs and complexity, and faster time to market

Quix Streams and InfluxDB are open-source systems you can self-manage (if that is your preference). However, InfluxDB can be used as a fully managed service (InfluxDB Cloud). Similarly, you have the option of deploying Quix Streams applications to Quix Cloud, a fully managed cloud platform. Using InfluxDB Cloud and Quix Cloud brings several advantages compared to the self-managed route:

• No need for extensive infrastructure setup and maintenance
• Significantly reduced DevOps, financial, and operational burden
• You are free to focus on innovating, building, and releasing new features and capabilities faster

{{testimonial_Christoph-Dietrich}}

Scalable, reliable, future-proof data infrastructure

InfluxDB is arguably the most popular time series database available today. It's a great option to store time series data, regardless of volume. Customers value its reliability, scalability and proven performance.  

Quix shares these characteristics. Built by Formula 1 engineers and in production with Formula 1 teams, Quix is a robust solution that’s optimized to handle high-volume, high-velocity time series data:

• Highly scalable, Quix leverages Kafka and Kubernetes to provide data partitioning, consumer groups, and state management
• Reliable data delivery and failure recovery through exactly-once processing, data and service replication, changelogs, and checkpointing
• Highly available — Quix Cloud guarantees 99.99% uptime
• Able to process gigabytes of data per second, with consistently low latencies (in the double-digit millisecond range)

By pairing InfluxDB and Quix, you end up with a stable, dependable system that’s ready to store, analyze, and leverage time series data at any scale.

{{testimonial_Fernando-Ayuso}}

What kind of use cases can I enable with InfluxDB and Quix?

By leveraging InfluxDB as your time series database and Quix as your stream processing engine, you can extract real-time analytics from time series data, and flexibly manipulate time series data as you see fit. Here are some examples of what you can achieve with Quix and InfluxDB:

• Capture and analyze network data and server metrics to detect anomalies and trends
• Collect and analyze data points generated by manufacturing machinery for predictive maintenance purposes
• Gather and analyze sensor data from vehicles to monitor car performance and trigger alerts
• Ingest and analyze financial transactions to detect patterns indicative of fraud
• Downsample time series data to reduce storage costs
• Filter and sync time series data from InfluxDB v2 to InfluxDB v3