30 Nov, 2021 | Explainer

How does serverless compute work in stream processing?

Learn more about the infrastructure that accelerates building data driven-products. We break it down with a super-speedy explanation and video.

Steve Rosam
Words by
Steve Rosam, Full-stack developer
145 feature image

With ready-to-use infrastructure, you can jump straight in to writing code

In the journey toward deploying a data-driven product to production, there are a lot of hurdles that a developer or data scientist must clear.

Our team recently charted these, describing each step in a typical CI/CD (continuous integration/continuous delivery) development pipeline. Within each process, we found additional sub-processes that slowed development even more.

Sketching out our customers’ experience with traditional software development tools energized our team at Quix because we’re working to eliminate the complexity and hassle associated with stream processing.

That’s why I wanted to bring you a short explainer of Quix’s serverless compute for stream processing. Our platform integrates Git source code, Docker, Kafka and Kubernetes — so you can focus on writing code rather than standing up a development environment.

Serverless compute for stream processing with Python

Want to see what that looks like? Here’s a short, 75-second video I made that walks you through these elements Quix:

A few highlights you’ll see:

  • Quix manages an elastic environment for you. You don’t need to worry about servers, nodes, memory or CPU. Set the limits for your deployment, and Quix will do the rest.
  • You can commit and track changes using GIT as the underlying source code repository
  • Docker integration and the docker file supplied with each project allow you to fine-tune things if needed
  • You can view the build and runtime logs
  • Deploy dashboards and public APIs as part of your application with the public access options

How to deploy serverless compute for data stream processing

As a next step, you can take a closer look at deploying to the Quix serverless environment. Here’s an 80-second video to show how I do that for a very simple website:

To deploy code, select any variables needed, choose the version and desired resources, and hit deploy. Your code will be built, deployed and run. If you want to deploy a frontend or any publicly accessible models with API or application code, you’ll also want to enable Public Access on the Network tab.

Deployments can be either a “job” or a “service.” Jobs are for training ML models or one-time operations, such as batch importing historic data. Jobs are marked as completed after the process terminates.

Services are for running application code, ML models or frontends continuously in production. Services are automatically restarted if the process terminates.

I hope you enjoyed my short walk-throughs. Curious to try it yourself? Sign up for free or explore our YouTube channel for more explainers and video tutorials. If you have any questions, join us on our community Slack.


Talk to a technical expert about your use case if you’re considering using stream processing in your business.

Book a demo
Steve Rosam
words by
Steve Rosam, Full-stack developer

Steve Rosam is a Full-stack developer at Quix, where he creates and maintains solutions both in-house and for customers. Steve has worked as a software developer for two decades, previously in a variety of industries including automotive, finance, media and security.

Previous Post Next Post

Related content

View all
SQL CDC feature
Explainer | 9 Nov, 2022
Build a CDC pipeline with the Quix SQL Server connector
Create a CDC pipeline and publish data to Kafka topics in just a few minutes with our open source SQL Server connector.
Steve Rosam
words by
Steve Rosam, Full-stack developer
Blog 183 feature
Explainer | 23 Aug, 2022
Why industrial IoT is essential and how to implement it
The internet of things has expanded from small personal devices to warehouses and factories. This post will look at how IIoT impacts various industries and how to start or accelerate your transformation.
words by
Mike Rosam, CEO & Co-Founder
How to capture and store time series data
Explainer | 9 Aug, 2022
Four solutions for handling time series data
Most data in streaming applications such as IoT, finance, user behavior analysis and automotive is time-series data. Learn how to capture, process and apply it to get the most value from it.
Tomas Neubauer
words by
Tomáš Neubauer, CTO & Co-Founder

The Stream

Updates to your inbox

Get the data stream processing community's newsletter. It's for sharing insights, events and community-driven projects.

Background image