29 Sep, 2022 | Community

The Stream — September 2022 edition

The September 2022 edition of The Stream: covering this month in stream processing on the internet.

Words by
Mike Rosam, CEO & Co-Founder
The Stream September 2022

Stream processing is defining a new world for organizations that want to be able to use and make sense of data in the moment it is created. Unfortunately, this leading-edge technology is not easily and broadly understood, and even less easily wrangled into a cohesive, performant data pipeline.

In the Gartner market survey, “Adopt Stream Data Integration to Meet Your Real-Time Data Integration and Analytics Requirements,” the research company found that “In our annual survey for the data integration tools market, 47% of organizations reported that they need streaming data to build a digital business platform, yet only 12% of those organizations reported that they currently integrate streaming data for their data and analytics requirements.”

That means that just 5.6% of companies are actually harnessing the power of streaming data to generate immediate insightspower ML capabilities or feed fresh data back into their products. Looking at this on the adoption curve, this represents just the beginning: the so-called “innovators.”

The next chunk of companies to use stream processing — the 41.4% of companies who would fit either into an “early adopters” or “early majority” category — are aware they want and need to be able to use this technology, but can’t easily do so.

It’s tricky. Combining Kafka, Kubernetes, and many other newer technologies is not for the faint of heart (or light of wallet). Large-scale companies that have already deeply invested in developer resources here such as John and Alibaba are leading the charge, but where does that leave midmarket and digital-first companies with fewer resources?

This month, we’re connecting with organizations of all sizes and industries by hosting community meetups in London, Berlin, Munich and Austin, speaking at events like Big Data London, and joining the Kafka user community at Current in Austin in early October. Our goal is to bring the power of stream processing to everyone by building shared knowledge, understanding and great tools to democratize data stream processing. Join us.

Streaming first infrastructure for real time ML

Streaming-first infrastructure for real-time machine learning

A streaming infrastructure can improve ML prediction latency and continual learning. Chip Huyen and Anthony Alford explain why an event-driven microservices architecture is a better choice for using continual learning than a REST-based architecture.

Learn more
Four open source stream processing technologies

Explore four popular open-source stream processing technologies

Gary Stafford offers a technical two-part post comparing these stream processing projects: Apache Spark Structured Streaming, Apache Kafka Streams, Apache Flink, and Apache Pinot.

Explore more →

Zhamak Dehghani Big Data LDN

Understanding the Data Mesh with Zhamak Dehghani

Dig into one of the most talked about trends in data, the Data Mesh. Dehghani offers “atomic steps to rewire the sociotechnical backbone of your organization,” and how to apply it to get value from data rapidly, sustainably and at scale.

Read more →

Current Austin

Destination: Austin, Texas for Current

It’s not just the smell of brisket in the air — it’s the Kafka user community coming together at Current, the conference for data professionals to learn about data streaming, Oct. 4-5.

Register now →

More insights

  • According to Memgraph, the most common use case for stream processing is monitoring sensor or device data.
  • In the latest Redpanda University course, you’ll learn about stateless and stateful stream processing and work through an advanced end-to-end tutorial. Enroll for free.
  • Ververica offers a new video on streaming concepts for Flink: “Exactly Once Fault Tolerance Guarantees.” Watch on YouTube.
  • What’s next for streaming analytics on Spark and Flink? Mike Ridley discusses complex event processing with Macrometa on YouTube.

Join The Stream community, where you’ll find developers, engineers and scientists supporting each other while working on streaming projects.

Join us
words by
Mike Rosam, CEO & Co-Founder

Mike Rosam is Co-Founder and CEO at Quix, where he works at the intersection of business and technology to pioneer the world's first streaming data development platform. He was previously Head of Innovation at McLaren Applied, where he led the data analytics product line. Mike has a degree in Mechanical Engineering and an MBA from Imperial College London.

Previous Post Next Post

Related content

View all
Quix streams blog feature image
Community | 2 Mar, 2023
Introducing Quix Streams, an open source library for telemetry data streaming
Lightweight, powerful, no JVM and no need for separate clusters of orchestrators. Here’s a look at our next-gen streaming library for C# and Python developers including feature summaries, code samples, and a sneak peek into our roadmap.
Tomas Neubauer
words by
Tomáš Neubauer, CTO & Co-Founder
The Stream February 2023
Community | 28 Feb, 2023
The Stream — February 2023 edition
Build a simple event-driven system to get ML predictions with Python and Apache Kafka
words by
Mike Rosam, CEO & Co-Founder
The Stream January 2023
Community | 25 Jan, 2023
The Stream — January 2023 edition
How can you send time-series data to Apache Kafka using Python and Pandas? Plus Apache Flink news, memes, and meetups
words by
Mike Rosam, CEO & Co-Founder

The Stream

Updates to your inbox

Get the data stream processing community's newsletter. It's for sharing insights, events and community-driven projects.

Background image