There’s no instruction manual for stream processing. With established technology, there are volumes of information about how to do something better — books, blogs, user groups, reference architectures. But with an emerging technology as complex and varied as data stream processing, it’s a brave new world.
That’s why Quix is sponsoring the first community dedicated to figuring this all out. We’re calling it The Stream. Whether you’ve been processing data on a message broker like Kafka for a while, or you’re simply stream-curious, you’re welcome. You’ll find friends and helpful allies here.
Our community connects on Slack and in person at meetups. Our first two live events are in Berlin this month. You should be a part of the community if you:
Work with data — whether as a software developer, data scientist, or engineer of another stripe. We’re having great conversations with mechanical and electrical engineers who focus on building IoT devices and need the data stream processing backbone to make them go.
Write code in Python — while many data-oriented technologies have centered on SQL, and streaming data has roots in Java and Scala for software engineering, we’re focusing on Python because it’s easy to use (making streaming accessible to newcomers), extensive (there’s a huge ecosystem of packages, like Pandas, that do much of the work for you) and incredibly flexible (making streaming useful to anyone from any industry).
Are interested in real-time data — you recognize the power and potential of streaming data, migrating workloads from batch to stream processing, or realizing the benefits of it such as lower latency, more efficient use of compute resources, and real-time automation or ML.
I believe passionately in streaming data. It will be the defining technology of this era. The 2010s focus on big data is giving way to faster architectures that enable you to build new product categories.
It’s early days, and innovation can be complicated. I believe this community will be coming together to better understand not only how to use stream processing, but also when not to. We’re not about technology for technology’s sake. We’re about finding new and better ways to handle the massive volume and velocity of data that’s streaming in from IoT devices and digital experiences every day.
Our community’s goals are learning, supporting each other, and helping to develop connected devices and data-driven products faster, easier and more efficiently.
You might use Quix for that. You might not. Either way, you’re welcome and we hope to make the community a valuable forum and connection for you.
Start today on the Slack channel and introduce yourself. Drop a meeting on my calendar to talk about your use case. I’m excited by the potential our community can collectively realize. Join us!
Meet The Stream in person
We’re gathering in Berlin on March 31, 2022 to talk about the ins and outs of stream processing.
Can better data help policy makers and communities build resilience?
How can streaming data help fight against food insecurity? “By using machine learning and other tools, high-frequency data collected by MIRA has greater potential to predict food insecurity outcomes and identify households likely to be most affected by shocks and stressors.”
A glossary for stream processing
We’ve put together a long list of terms and definitions important to stream processing. You won’t find anything specifically about Quix in there; it’s as wide-reaching and inclusive as possible to help anyone navigate the world of real-time data.
More insights
- Data Council offers a “100% no bullsh*t guarantee” event about data science, engineering and analytics. You’ll want to join for the easy networking and the abundance of speakers across six learning tracks. There’s something for everyone here.
- The European Union has proposed an addition to its digital rulebook. Read about it on TechCrunch.
- Niyi Odumosu, Associate Solutions Architect at Confluent, uses the Metrics API, Docker, Prometheus, Grafana, Splunk and Datadog to put together a full monitoring solution for your Confluent Cloud deployment.
- To settle some confusion about the terms real time and stream processing, check out How stream processing goes beyond real-time.
Mike Rosam is Co-Founder and CEO at Quix, where he works at the intersection of business and technology to pioneer the world's first streaming data development platform. He was previously Head of Innovation at McLaren Applied, where he led the data analytics product line. Mike has a degree in Mechanical Engineering and an MBA from Imperial College London.
