September 29, 2022
Industry insights

The Stream — September 2022 edition

The September 2022 edition of The Stream: covering this month in stream processing on the internet.

The Stream September 2022 banner.
Quix Streams combines an Apache Kafka client with a stream processing library, offering a feature-rich pure Python alternative to the kafka-python library. Deploy on your stack or on Quix Cloud for scalable, stateful and fault-tolerant ETL without the headache of cross-language debugging.

👉 Star the repo on GitHub and pip install quixstreams to get started.

Stream processing is defining a new world for organizations that want to be able to use and make sense of data in the moment it is created. Unfortunately, this leading-edge technology is not easily and broadly understood, and even less easily wrangled into a cohesive, performant data pipeline.

In the Gartner market survey, “Adopt Stream Data Integration to Meet Your Real-Time Data Integration and Analytics Requirements,” the research company found that “In our annual survey for the data integration tools market, 47% of organizations reported that they need streaming data to build a digital business platform, yet only 12% of those organizations reported that they currently integrate streaming data for their data and analytics requirements.”

That means that just 5.6% of companies are actually harnessing the power of streaming data to generate immediate insights, power ML capabilities or feed fresh data back into their products. Looking at this on the adoption curve, this represents just the beginning: the so-called “innovators.”

The next chunk of companies to use stream processing — the 41.4% of companies who would fit either into an “early adopters” or “early majority” category — are aware they want and need to be able to use this technology, but can’t easily do so.

It’s tricky. Combining Kafka, Kubernetes, and many other newer technologies is not for the faint of heart (or light of wallet). Large-scale companies that have already deeply invested in developer resources here such as John Deere, and Alibaba are leading the charge, but where does that leave midmarket and digital-first companies with fewer resources?

This month, we’re connecting with organizations of all sizes and industries by hosting community meetups in London, Berlin, Munich and Austin, speaking at events like Big Data London, and joining the Kafka user community at Current in Austin in early October. Our goal is to bring the power of stream processing to everyone by building shared knowledge, understanding and great tools to democratize data stream processing. Join us.

An image of a cat along tech company logos.

Streaming-first infrastructure for real-time machine learning

A streaming infrastructure can improve ML prediction latency and continual learning. Chip Huyen and Anthony Alford explain why an event-driven microservices architecture is a better choice for using continual learning than a REST-based architecture.

Learn more

Logos of four open source stream processing technologies.

Explore four popular open-source stream processing technologies

Gary Stafford offers a technical two-part post comparing these stream processing projects: Apache Spark Structured Streaming, Apache Kafka Streams, Apache Flink, and Apache Pinot.

Explore more →

Zhamak Dehghani Big Data conference.

Understanding the Data Mesh with Zhamak Dehghani

Dig into one of the most talked about trends in data, the Data Mesh. Dehghani offers “atomic steps to rewire the sociotechnical backbone of your organization,” and how to apply it to get value from data rapidly, sustainably and at scale.

Read more →

Colorful event banner saying Current.

Destination: Austin, Texas for Current

It’s not just the smell of brisket in the air — it’s the Kafka user community coming together at Current, the conference for data professionals to learn about data streaming, Oct. 4-5.

Register now →

More insights

  • According to Memgraph, the most common use case for stream processing is monitoring sensor or device data.
  • In the latest Redpanda University course, you’ll learn about stateless and stateful stream processing and work through an advanced end-to-end tutorial. Enroll for free.
  • Ververica offers a new video on streaming concepts for Flink: “Exactly Once Fault Tolerance Guarantees.” Watch on YouTube.
  • What’s next for streaming analytics on Spark and Flink? Mike Ridley discusses complex event processing with Macrometa on YouTube.

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Related content

Featured image for the "Navigating stateful stream processing" post published on the Quix blog
Industry insights

Navigating stateful stream processing

Discover what sets stateful stream processing apart from stateless processing and read about its related concepts, challenges and use cases.
Tim Sawicki
Words by
windowing in stream processing
Industry insights

A guide to windowing in stream processing

Explore streaming windows (including tumbling, sliding and hopping windows) and learn about windowing benefits, use cases and technologies.
Daniil Gusev
Words by
real time feature engineering architecture diagram
Industry insights

What is real-time feature engineering?

Pre-computing features for real-time machine learning reduces the precision of the insights you can draw from data streams. In this guide, we'll look at what real-time feature engineering is and show you a simple example of how you can do it yourself.
Tun Shwe
Words by