back
July 18, 2023
|
Ecosystem

Accelerating AI-ready application development: Quix and Confluent partnership

Teams can now build AI applications on Confluent’s data in motion, with Quix, the AI-ready event streaming application framework.

Quix ML model icons on black background.
Quix brings DataFrames and the Python ecosystem to stream processing. Stateful, scalable and fault tolerant. No wrappers. No JVM. No cross-language debugging.

Our library is open source—support the project by starring the repo.

Introduction

Confluent will go down in history as the company that set the world’s data in motion. We have long admired this mission having experienced the power of Apache Kafka at McLaren Racing where we used it to gain an advantage from real-time data processing.

At McLaren we ingested streaming data into Kafka fairly quickly, but struggled to roll out the technology across the business due to the language barriers and impedance gaps inherent in modern, multi-disciplinary technical organisations. Ultimately we spent three years building an internal developer platform around Kafka, and still burned hundreds of hours supporting developers across the team to build event streaming applications.

We founded Quix to solve these problems by building a tool that empowers AI teams to develop real-time applications that run directly on Kafka without complex infrastructure build-outs or high-touch support from platform teams.

Today, I’m delighted to announce the launch of a deep integration between Quix and Confluent. Together with today’s announcement that Quix is a launch partner in the Connect with Confluent programme, we fill a crucial gap in the AI ecosystem and take a giant leap on our mission to empower developers to unleash the full potential of real-time applications.

Why Quix and Confluent are better together

Confluent is the de facto standard in streaming data. However, until now, one critical capability has been missing from the ecosystem—the ability for teams to rapidly develop AI applications directly on Kafka. This is where Quix steps in, offering an event streaming application framework that empowers your teams to unlock the full potential of real-time data.

| | Confluent | Quix | Together | |-------------------|-------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| | Connect | ✅ 180+ ready-to-run Kafka Connect connectors with fully managed clusters. | ✅ 60+ open source code samples running in Fully managed Kubernetes containers. Compatible with any framework e.g. Airbyte, Debezium MQTT, OpenTelemetry etc. | 🔥 Quickly ingest and sink any data from any source to any destination | | API | ✅ Confluent REST Proxy API (producer/consumer) Confluent Cloud API Connect REST API ksqlDB REST API Metadata API Schema Registry API | ✅ REST producer/consumer API WebSocket producer/consumer API REST Query API REST Portal API | 🔥 Rapidly integrate to any external system e.g. Jupyter, MLFlow, Weights & Biases, Seldon etc. | | Streaming | ✅ Enterprise ready fully managed Kafka | 🔐 Bring Your Own Kafka with native TCP/IP integration | 🔥 Stream and process data with best-in-class tooling | | SQL Processing | ✅ Fully managed ksqlDB (and FlinkSQL coming Q4 2023) clusters | ❌ None | 🔥 Transform and join data in motion | | Python Processing | ❌ None | ✅ Fully managed Kubernetes clusters with Quix Streams open source Python library. | 🔥 Real-time machine learning feature computation and AI model serving | | Storage | ✅ Unlimited tiered storage for long-term Kafka log persistence | ✅ Queryable storage, replay service and data exporter for AI development and testing | 🔥 Access streaming data for exploratory data analysis and build traceable AI application pipelines | | KafkaOps tools | ✅ Management console. Cluster Replicator. Schema registry. Stream Lineage. Stream Catalog. | ❌ Application config and monitoring tools only | 🔥 Control Kafka in dev and production environments | | DevOps tools | ❌ ksqlDB pipeline builder only. | ✅ Pipeline builder. Samples library. Cloud IDE. Git integration, YAML IaC and CI/CD workflow | 🔥 Enable data science and ML teams to quickly build AI-ready applications |

Demo: see how the integration works

Conclusion

The partnership between Quix and Confluent represents a significant milestone in the world of real-time applications. By combining the power of Confluent's industry-leading Kafka platform with Quix's F1-derived event streaming application framework, organisations can now build AI-ready applications directly on data in motion in Confluent Cloud. Explore the new use cases you can unlock with Confluent and Quix or try it yourself here.

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Related content

Banner image for the article "Understanding Kafka’s auto offset reset configuration: Use cases and pitfalls" published on the Quix blog
Ecosystem

Understanding Kafka’s auto offset reset configuration: Use cases and pitfalls

The auto.offset.reset configuration defines how Kafka consumers should behave when no initial committed offsets are available for the partitions assigned to them. Learn how to work with this configuration and discover its related challenges.
Tim Sawicki
Words by
kafka vs spark logos
Ecosystem

Kafka vs Spark - a comparison of stream processing tools

This comparison specifically focuses on Kafka and Spark's streaming extensions — Kafka Streams and Spark Structured Streaming. Kafka Streams excels in per-record processing with a focus on low latency, while Spark Structured Streaming stands out with its built-in support for complex data processing tasks, including advanced analytics, machine learning and graph processing.
Tun Shwe
Words by
image with aws fargate and lambda logos
Ecosystem

Fargate vs Lambda: a comparison of serverless technologies

The main difference between these two serverless compute platforms is that AWS Fargate takes care of the underlying VMs, networking, and other resources you need to run containers using ECS or EKS, whereas AWS Lambda lets you run standalone, stateless functions without having to consider any of the infrastructure whatsoever.
Mike Rosam
Words by