back
September 29, 2022
|
Industry insights

The Stream — September 2022 edition

The September 2022 edition of The Stream: covering this month in stream processing on the internet.

The Stream September 2022 banner.

Python stream processing, simplified

Pure Python. No JVM. No wrappers. No cross-language debugging. Use streaming DataFrames and the whole Python ecosystem to build stream processing applications.

Python stream processing, simplified

Pure Python. No JVM. No wrappers. No cross-language debugging. Use streaming DataFrames and the whole Python ecosystem to build stream processing applications.

Data integration, simplified

Ingest, pre-process and load high volumes of data into any database, lake or warehouse, without overloading your systems or budgets.

The 4 Pillars of a Successful AI Strategy

Foundational strategies that leading companies use to overcome common obstacles and achieve sustained AI success.
Get the guide

Guide to the Event-Driven, Event Streaming Stack

Practical insights into event-driven technologies for developers and software architects.
Get the guide
Quix is a performant, general-purpose processing framework for streaming data. Build real-time AI applications and analytics systems in fewer lines of code using DataFrames with stateful operators and run it anywhere Python is installed.

Stream processing is defining a new world for organizations that want to be able to use and make sense of data in the moment it is created. Unfortunately, this leading-edge technology is not easily and broadly understood, and even less easily wrangled into a cohesive, performant data pipeline.

In the Gartner market survey, “Adopt Stream Data Integration to Meet Your Real-Time Data Integration and Analytics Requirements,” the research company found that “In our annual survey for the data integration tools market, 47% of organizations reported that they need streaming data to build a digital business platform, yet only 12% of those organizations reported that they currently integrate streaming data for their data and analytics requirements.”

That means that just 5.6% of companies are actually harnessing the power of streaming data to generate immediate insights, power ML capabilities or feed fresh data back into their products. Looking at this on the adoption curve, this represents just the beginning: the so-called “innovators.”

The next chunk of companies to use stream processing — the 41.4% of companies who would fit either into an “early adopters” or “early majority” category — are aware they want and need to be able to use this technology, but can’t easily do so.

It’s tricky. Combining Kafka, Kubernetes, and many other newer technologies is not for the faint of heart (or light of wallet). Large-scale companies that have already deeply invested in developer resources here such as John Deere, Booking.com and Alibaba are leading the charge, but where does that leave midmarket and digital-first companies with fewer resources?

This month, we’re connecting with organizations of all sizes and industries by hosting community meetups in London, Berlin, Munich and Austin, speaking at events like Big Data London, and joining the Kafka user community at Current in Austin in early October. Our goal is to bring the power of stream processing to everyone by building shared knowledge, understanding and great tools to democratize data stream processing. Join us.

An image of a cat along tech company logos.

Streaming-first infrastructure for real-time machine learning

A streaming infrastructure can improve ML prediction latency and continual learning. Chip Huyen and Anthony Alford explain why an event-driven microservices architecture is a better choice for using continual learning than a REST-based architecture.

Learn more

Logos of four open source stream processing technologies.

Explore four popular open-source stream processing technologies

Gary Stafford offers a technical two-part post comparing these stream processing projects: Apache Spark Structured Streaming, Apache Kafka Streams, Apache Flink, and Apache Pinot.

Explore more →

Zhamak Dehghani Big Data conference.

Understanding the Data Mesh with Zhamak Dehghani

Dig into one of the most talked about trends in data, the Data Mesh. Dehghani offers “atomic steps to rewire the sociotechnical backbone of your organization,” and how to apply it to get value from data rapidly, sustainably and at scale.

Read more →

Colorful event banner saying Current.

Destination: Austin, Texas for Current

It’s not just the smell of brisket in the air — it’s the Kafka user community coming together at Current, the conference for data professionals to learn about data streaming, Oct. 4-5.

Register now →

More insights

  • According to Memgraph, the most common use case for stream processing is monitoring sensor or device data.
  • In the latest Redpanda University course, you’ll learn about stateless and stateful stream processing and work through an advanced end-to-end tutorial. Enroll for free.
  • Ververica offers a new video on streaming concepts for Flink: “Exactly Once Fault Tolerance Guarantees.” Watch on YouTube.
  • What’s next for streaming analytics on Spark and Flink? Mike Ridley discusses complex event processing with Macrometa on YouTube.

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Related content

Banner image for the article "Rethinking Build vs Buy" published on the Quix blog
Industry insights

Rethinking “Build vs Buy” for Data Pipelines

“Build vs buy” is outdated — most companies need tools that provide the flexibility of a build with the convenience of a buy. It’s time for a middle ground.
Mike Rosam
Words by
Banner image for the article "When a European Manufacturing Leader Needed to Modernize Their Data Stack" published on the Quix blog
Industry insights

When a European manufacturing leader needed to modernize their data stack

Learn how an industrial machinery provider went from processing their sensor data in batches to real time using Python and Quix Streams instead of Flux..
Tun Shwe
Words by
Banner image for the article "How to Empower Data Teams for Effective Machine Learning Projects" published on the Quix blog
Industry insights

How to Empower Data Teams for Effective Machine Learning Projects

Learn how to boost success rates ML projects by empowering data teams through a shift-left approach to collaboration and governance.
Mike Rosam
Words by