The Stream — January 2022 edition
The January 2022 edition of The Stream: covering this month in stream processing on the internet.
What’s a data-driven company if it doesn’t take an opportunity to drive its content with data? Instead of filling our January newsletter with things we think you should know, we turned back the clock to 2021 and considered what other people like you — data engineers, data scientists and developers — want to know.
The data revealed our most popular tutorials, explainer blog posts, research reports and how-to guides. It told us that the data engineering community is ravenous to learn how to streamline data. They’ve realized that dumping raw data into a warehouse for cleanup and query later makes everyone’s jobs harder than they need to be.
The problem is that it’s complicated. Until now, no one except an elite handful of tech juggernauts had harnessed the power of stream processing because it took a literal army of engineers to do it.
We think the future of data is stream processing for everyone. Soon, you’ll see businesses of every size processing data in memory, on a message broker like Kafka to extract its value before their data gets sunk into a lake or buried under ever-increasing volumes of data in a warehouse.
Based on our reader data, you think so, too. Many of you searched us out for our tutorials on doing real-time stream processing with Kafka and Python. Many also looked for our research and head-to-head comparison on how well Python-based stream processing client libraries perform.
So, without further ado, here’s our digest of the top tutorials, explainers and research reports from 2021 that were most helpful to our users.
Kafka + Python = %$#&?!
Python gets the most love from data scientists and other data-friendly developers, but when it comes to Kafka, Python gets the cold shoulder. Here’s how they work together.
Why is streaming data so hard to handle?
And why aren’t these difficulties already solved? Our CTO explains in not-too-technical terms why stream processing has been out of reach for most organizations — until now.
An analogy is worth 1,000 words →
Which Python library is best for stream processing?
When you build a product, you do research — exhaustive competitive research. Our CEO offers a deep-deep-dive on Spark vs. Flink vs Quix performance.
Everything you wanted to know about Kafka but were afraid to ask
Kafka isn’t just tricky technology — it’s tricky terminology. Dig into what makes Kafka different, when you should (and shouldn’t) use it, and how it works.
More insights
- An organization’s ability to access customer data, and the demand to build engines that make decisions about how to process it, are the key challenges for customer-centric, data-driven growth, according to McKinsey.
- Levi Strauss and John Deere are among the “old” companies pioneering stream processing. How? They’re empowering employees to take advantage of stream processing and ML.
- You got stream processing to work. Now how do you get it to scale? Insights from Alibaba, Twitter and more.
- Read more of our most popular articles from 2021, complete with key takeaways.
What’s a Rich Text element?
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
Static and dynamic content editing
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
How to customize formatting for each rich text
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
Check out the repo
Our Python client library is open source, and brings DataFrames and the Python ecosystem to stream processing.
Mike Rosam is Co-Founder and CEO at Quix, where he works at the intersection of business and technology to pioneer the world's first streaming data development platform. He was previously Head of Innovation at McLaren Applied, where he led the data analytics product line. Mike has a degree in Mechanical Engineering and an MBA from Imperial College London.
Related content
The challenges of processing data from devices with limited connectivity and how to solve them
Rethinking “Build vs Buy” for Data Pipelines
How to Empower Data Teams for Effective Machine Learning Projects