Integrate and process real-time data at scale with Quix, Google Pub/Sub & Python

How do Quix and Google Pub/Sub work together?

Google Pub/Sub is the real-time messaging service component. It has the following responsibilities:

• Collect data from source systems
• Publish messages to Quix, the data processing component

Quix is the Python stream processor, and it serves the following purposes:

• Ingest messages from Google Pub/Sub topics
• Process received messages 
• Send transformed data to destination systems (via Quix connectors) so it can be stored and operationalized
• Use data received from Google Pub/Sub to power real-time capabilities

Together, Google Pub/Sub and Quix offer a complete, end-to-end solution that can integrate, process, and extract value from real-time data. 

Integrating Quix and Google Pub/Sub only takes a few clicks. This is made possible by the Quix external source connector, which enables you to send data from any given topic in Google Pub/Sub into a Quix-hosted Apache Kafka topic. 

Once the integration is in place and data flows from Google Pub/Sub to Quix, you can implement your data processing logic. You do so via  Quix Streams, an open-source technology that combines an Apache Kafka client with a Python stream processing library. Here are its key capabilities:

• Easy to learn. Streaming DataFrame API (similar to pandas DataFrame) for tabular data transformations.
• Windowing. Aggregations over hopping and tumbling windows (sliding windows are on the roadmap).
• Stateless processing. This includes grouping, filtering, projections, and dropping columns. 
• Custom processing. The ability to transform data and implement custom, user-defined Python functions.
• Stateful operations. Stateful processing using RocksDB to persist state.
• Simple output routing. The ability to succinctly produce data to output Kafka topics using the to_topic() method.
• Flexible processing guarantees. At-least-once and exactly-once processing semantics.
• Versatile serialization. Support for various data serialization formats: bytes, string, integer, double, JSON, Protobuf, and Avro.
• Simplified deployment. Seamless integration with Quix Cloud, a fully managed platform offering a straightforward way to deploy and manage your stream processing applications.

To learn more about Quix Streams and how to use its features, see the Quix Streams tutorials.  

What are the benefits of using Quix alongside Google Pub/Sub?

Pure Python development experience

Google Cloud Run, Dataproc, and Dataflow are examples of Google Cloud Platform services you might use to process messages from Google Pub/Sub topics in real time. However, neither of them is specifically designed to serve Python users.  

In contrast, Quix is a stream processor that’s tailor-made for Python developers. 

Compared to the GCP stream processing solutions, Quix offers a unique blend of advantages:

• Pure Python coding and debugging experience
• Intuitive Streaming DataFrame API with a modern Python syntax and a gentle learning curve (especially if you’re familiar with pandas)
• A straightforward way to integrate all your favorite Python libraries into your workflow (scikit-learn, TensorFlow, PyTorch, etc)
• There’s no server-side engine

{{testimonial_Ben-Gamble}}

Flexible, comprehensive tooling

Google Pub/Sub makes it convenient to stream data at scale by offering supporting tools like robust monitoring capabilities, configurable publish and subscribe options (e.g., message compression and flow control), and even an emulator to develop and test your applications locally. 

Similarly, Quix offers everything you need to easily and conveniently build, deploy, and manage industrial-strength stream processing applications:

• CI/CD support. Integrations with any Git provider (e.g., GitHub, Bitbucket, Azure DevOps) for seamless CI/CD processes.
• Environment control. Multiple projects and environments (linked to Git) for streamlined environment management.
• Team collaboration. Multi-user collaboration at project and environment levels through organization and permission management.
• Infrastructure management. Infrastructure as code (IaC) using Quix YAML (similar to Helm charts) with automated synchronization.
• Observability and monitoring. Real-time logs, metrics, data explorers, and waveform and table views.
• Security. Securely manage secrets and sensitive information.
• Dev tools. Online code editor, Python code templates, and pre-built connectors for various data sources and sinks (e.g., MQTT, InfluxDB, Redis).
• Pipeline management. Functionality to scale resources, adjust replicas, and manage CPU and memory for your streaming application.
• Rapid prototyping. An in-built Quix-hosted Apache Kafka broker for testing and fast prototyping.
• Local development. CLI tool to create, debug, and run your pipeline locally, and then quickly deploy it to the cloud. 

{{testimonial_Carey-McLean}}

Reduced costs and complexity, and faster time to market

Google Cloud Pub/Sub is a fully managed messaging service. Quix is also a fully managed solution. Together, they remove the complexity of handling real-time data pipelines in-house: 

• There’s no need to worry about configuring and managing streaming infrastructure
• Predictable costs and significantly reduced DevOps, financial, and operational burden
• You are free to focus entirely on innovating, building, and releasing new features, products, and capabilities that rely on real-time data 

{{testimonial_Christoph-Dietrich}}

Proven scalability and reliability 

Due to its resiliency, high availability, scalability, low latency, and in-order message delivery, Google Pub/Sub is a trustworthy solution for data streaming. 

Quix is an equally dependable solution:

• Highly scalable, leveraging Kafka and Kubernetes under the hood to provide data partitioning, consumer groups, and state management
• Reliable data delivery and failure recovery through exactly-once processing, data and service replication, changelogs, and checkpointing
• Highly available — Quix Cloud guarantees 99.99% uptime
• Able to process billions of events per day, with consistently low latencies (in the double-digit millisecond range)

By combining Google Pub/Sub with Quix, you get a rock-solid, future-proof system that can stream and process real-time data at scale. You can trust it to grow with your business, providing the reliability and performance you need as your demands evolve. 

{{testimonial_Fernando-Ayuso}}

What kind of use cases can I enable with Google Pub/Sub and Quix?

By leveraging Google Pub/Sub as your messaging middleware and Quix as your Python stream processor, you can deliver event-driven architectures, real-time data pipelines, and streaming applications.

Here are some concrete examples of what you can achieve by pairing Quix and Google Pub/Sub:

• Real-time fraud & cheat detection
• Real-time sentiment & clickstream analysis
• Real-time predictive maintenance
• Live dashboards
• Real-time content recommendations
• Streaming ETL workflows and real-time ML products