Three ways to get your data into Quix

The Quix platform enables users to build and maintain stream processing applications at any scale. The first step to doing this is getting data into Quix. We know data comes from various sources, so the platform offers three ways of ingesting it: library connectors, a Software development kit (SDK) and APIs.

This blog post shows how to use all three methods. You’ll need to log into the Quix portal to get going. If you don’t have a free account, you can quickly sign up for one.

Using a pre-built connector is the fastest way to get data into Quix. The connectors expedite the process because they supply the foundational code, and users only need to provide the required values to deploy them.

Connectors in the Quix library are quick and easy ways of integrating with other platforms. They’re pre-built to cut down on development time while remaining customizable, so you can build precisely what you want in a fraction of the time.

We’ll look at an example that writes data from the Transport For London (TFL) BikePoint API.

• Navigate to the library and select the Python source filters from the left-hand menu
• Use the search bar to find the “TFL Bikepoint” library item
• Click “Setup & deploy”
• Enter your `tfl_primary_key` and `tfl_secondary_key`, which you get by signing up for a free TFL account
• Click “Deploy” in the dialog

And you’re off! A service will be deployed and started. You’ll be redirected to the home page, where you’ll see the running “TFL Bikepoint” service.
These few steps demonstrate how to start working with your data via a connector quickly. Search the library for connectors that accommodate different data types, and, if you don’t see one you’re looking for, write your own or try using the SDK or APIs.

The SDK lets you

• build sources, transformations and destinations
• write simple or complex data processing logic and connect it directly to any broker
• keep data in order by bundling it into streams,
• integrate various data sources and assemble a microservice data stack in either Python or C#

Let’s use the SDK now to stream some data into a topic.

To use the SDK to read a CSV file via Pandas data frames follows some of the same steps taken to ingest data from a library connector.

• Navigate to the library and select the Python source filters from the menu
• Use the search bar to find the “Write Car Data” library item
• Click the tile
• Click “Edit code” and “Save as project”

Instead of deploying a service, the code has been copied to a private Git repo just for you. You can now edit it and make it yours.

• The first line imports the Quix SDK: from quixstreaming, import QuixStreamingClient
• Then the output topic is opened: client.open_output_topic
• The stream is created: output_topic.create_stream()

These are all helpers that execute more lines of code behind the scenes. You can also replace these lines with something more specific to your use case, such as reading data from an HTTP endpoint or loading the data from your own CSV file.

When ready, run the code in the Quix online IDE by clicking “Run” in the top-right corner. You can also clone the code and work locally with your favorite development tools.

There are several APIs available to complement the SDK. These APIs allow you to connect platforms and technologies that don’t support the SDK. The Streaming Reader API, for example, allows data to be consumed in real time using WebSockets whereas the Streaming Writer APIs allow data to be written to the platform using both HTTP and WebSockets technologies.

To ingest data using an API, find an HTTP API sample in the library with the following steps:

• Navigate to the library and select the Javascript option under Languages
• Click the tile titled “HTTP API – Javascript”
• Click “Copy code”

You can now paste this code into a local editor or local file with a .html extension. When you open the page a new stream called “Hello World Stream” will be created and data will be streamed into the platform.

If you’d like a hand getting to know Quix, let us know! You can sign up for a demo or chat over Slack.