July 19, 2022

Three ways to get your data into Quix

When, why, and how to contribute code samples to the Quix library.

Quix platform banner.
Quix offers a pure Python framework for building real-time data pipelines. It's a Kafka client with a stream processing library rolled into one. No JVM, no cross-language debugging—just a simple Pandas-like API for handling streaming data. Deploy in your stack or on Quix Cloud for scalable, stateful, and fault tolerant stream processing.

Wondering where to start? Here are three options

The Quix platform enables users to build and maintain stream processing applications at any scale. The first step to doing this is getting data into Quix. We know data comes from various sources, so the platform offers three ways of ingesting it: library connectors, a Software development kit (SDK) and APIs.

This blog post shows how to use all three methods. You’ll need to log into the Quix portal to get going. If you don’t have a free account, you can quickly sign up for one.

Library connectors

Using a pre-built connector is the fastest way to get data into Quix. The connectors expedite the process because they supply the foundational code, and users only need to provide the required values to deploy them.

Connectors in the Quix library are quick and easy ways of integrating with other platforms. They’re pre-built to cut down on development time while remaining customizable, so you can build precisely what you want in a fraction of the time.

We’ll look at an example that writes data from the Transport For London (TFL) BikePoint API.

  • Navigate to the library and select the Python source filters from the left-hand menu
  • Use the search bar to find the “TFL Bikepoint” library item
  • Click “Setup & deploy”
  • Enter your `tfl_primary_key` and `tfl_secondary_key`, which you get by signing up for a free TFL account
  • Click “Deploy” in the dialog

And you’re off! A service will be deployed and started. You’ll be redirected to the home page, where you’ll see the running “TFL Bikepoint” service.
These few steps demonstrate how to start working with your data via a connector quickly. Search the library for connectors that accommodate different data types, and, if you don’t see one you’re looking for, write your own or try using the SDK or APIs.

The Quix SDK

The SDK lets you

  • build sources, transformations and destinations
  • write simple or complex data processing logic and connect it directly to any broker
  • keep data in order by bundling it into streams,
  • integrate various data sources and assemble a microservice data stack in either Python or C#

Let’s use the SDK now to stream some data into a topic.

To use the SDK to read a CSV file via Pandas data frames follows some of the same steps taken to ingest data from a library connector.

  • Navigate to the library and select the Python source filters from the menu
  • Use the search bar to find the “Write Car Data” library item
  • Click the tile
  • Click “Edit code” and “Save as project”

Instead of deploying a service, the code has been copied to a private Git repo just for you. You can now edit it and make it yours.

  • The first line imports the Quix SDK: from quixstreaming, import QuixStreamingClient
  • Then the output topic is opened: client.open_output_topic
  • The stream is created: output_topic.create_stream()

These are all helpers that execute more lines of code behind the scenes. You can also replace these lines with something more specific to your use case, such as reading data from an HTTP endpoint or loading the data from your own CSV file.

When ready, run the code in the Quix online IDE by clicking “Run” in the top-right corner. You can also clone the code and work locally with your favorite development tools.

Quix APIs

There are several APIs available to complement the SDK. These APIs allow you to connect platforms and technologies that don’t support the SDK. The Streaming Reader API, for example, allows data to be consumed in real time using WebSockets whereas the Streaming Writer APIs allow data to be written to the platform using both HTTP and WebSockets technologies.

To ingest data using an API, find an HTTP API sample in the library with the following steps:

  • Navigate to the library and select the Javascript option under Languages
  • Click the tile titled “HTTP API – Javascript”
  • Click “Copy code”

You can now paste this code into a local editor or local file with a .html extension. When you open the page a new stream called “Hello World Stream” will be created and data will be streamed into the platform.

No need to set up Quix alone — we’re here to help

If you’d like a hand getting to know Quix, let us know! You can sign up for a demo or chat over Slack.

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Related content

Graphic showing Quix Streams windowing code

Introducing Streaming DataFrames

Learn how Streaming DataFrames can simplify real-time data processing in Python with a familiar DataFrame approach.
Tomáš Neubauer
Words by
2.0 text on radial gradient background.

Announcing Quix 2.0—now with Git integration and multiple environments

Quix 2.0 is here 🚀 Designed around the concept of Infrastructure-as-Code, Quix 2.0 makes it easier to build and run reliable, powerful event-streaming applications that scale, with a single source of truth powered by Git.
Mike Rosam
Words by
Two black Quix windows open in different tabs.

Introducing Quix Streams, an open source library for telemetry data streaming

Lightweight, powerful, no JVM and no need for separate clusters of orchestrators. Here’s a look at our next-gen streaming library for C# and Python developers including feature summaries, code samples, and a sneak peek into our roadmap.
Tomáš Neubauer
Words by