23 May, 2022 | Explainer

How to build a no-code pipeline for sentiment analysis with our Snowflake connector

Three Quix connectors let you move data from Twitter to a Snowflake database while transforming it along the way. Learn how to set up the pipeline without writing any code.

Steve Rosam
Words by
Steve Rosam, Full-stack developer
Blog 189 feature

Spend less time integrating, and more time building

Connectors in the Quix library are quick and easy ways of integrating with other platforms. They’re pre-built to cut down on development time while remaining customizable, so you can build precisely what you want in a fraction of the time. Connectors also allow users to split their data processing pipeline into discrete microservices and manage all of them in the Quix platform. Building data stacks with microservices mean modularity and scalability for your product. You might want to swap tools or expand your product down the line. It allows you to exchange sources, transformations and destinations over time without rebuilding from scratch.

This tutorial will show you how to build a simple but complete data processing pipeline from scratch without touching any code. We’ll analyze tweets that contain information about Bitcoin and stream both raw and transformed data into Snowflake, a storage platform, using the Twitter, HuggingFace and Snowflake connectors. The pipeline you build will stream time-series, parameter data and metadata from Twitter to Snowflake — all managed from inside Quix.

I’ve made a video of this tutorial if you prefer watching to reading.

What you need for this tutorial

Step one: create a database

Sign in to your Snowflake account to create the Snowflake database to receive your data. Call this “demo data” and click “Create.” Snowflake will do the rest. You probably won’t see the database immediately, but it’ll be ready when you need it later.

Step two: get your data

In Quix, click into the library and search for the Twitter source connector. Click “Add new,” which adds the source to your pipeline and brings you back to the library.

You’ll notice that we need to fill in a few details for the connector to be deployed. Leave the output with the default value of “twitter-data.” If it isn’t pre-filled, make sure it is set to “twitter-data.”

Step two

Fill in your Twitter bearer token and search parameters. We’re using tweets as our data source, and we’ll listen specifically to Bitcoin tweets because of their high volume and would benefit from real-time processing. But you could change that to any search parameter.

Paste your Twitter Dev token and click “Deploy.” This deploys the pre-built Twitter connector. You’ll see that you already have some log output and the green dot that shows it’s running.

Step three: investigate the logs

To investigate the logs and look at your data, click on the Twitter connector tile on the home page. You can see that the tweets are being written into a stream in real time. You can see in the table view the string values, the tweet ID and the text.

Step three

Step four: transformation for sentiment analysis

In this step, you’ll do sentiment analysis on the tweets. Return to your workspace in Quix and click the “Add transformation” button. In the library, search for “HuggingFace.” We’ll use a HuggingFace model to determine sentiment analysis.

Click “Set up and deploy.” Choose “Twitter data” as the input topic. This is the raw data coming from Twitter. Set your output topic to hugging-face-output and use the default model. Leave all other values with their defaults.

Step four

Click “Deploy.” The pre-built connector will deploy on the Quix platform, and you’ll see the output shortly. You’ll be redirected to the home page to see that you now have two deployments up and running: the Twitter data source and the HuggingFace model.

Step five: delivery

In this step, you’ll deploy the Slowflake connector, which will stream the results of the HuggingFace model into your Snowflake database.

Click the “Add destination” button and search the library for the Snowflake connector.

Click “Set up and deploy” on the connector. You’ll need to fill in a few items here. Choose the hugging-face-output output topic, which will be the feed for this Snowflake sync.

Step five

To fill in the Snowflake locator and region (these are similar to a unique ID for your Snowflake instance), navigate to your Snowflake account. Copy the locator and region from the URL and paste them into the corresponding fields in the connector setup in Quix. Lastly, input your username and password.

Click “Deploy” on the Snowflake connector. If the credentials and connection details are correct, you’ll see the “Connected” status in the log and will be redirected to your workspace.

You can see that your transformed data is arriving in your database by going to your Snowflake dashboard and refreshing the page and expanding the tree view under your database. You should see all of your tables populated in this list. You’ll see event values, parameter values and metadata all populated by the Snowflake connector in Quix. Look for the sentiment score in the “parametervalues” table.

Setting sights on more extensive projects

Congratulations! You built a no-code pipeline that filters and collects data from Twitter, transforms it with a HuggingFace model and delivers it to a Snowflake database.

Source transformation destination

It’s easy to expand this example, thanks to the microservice approach used. Switch out sources, transformations and destinations or customize the ones you already have. The only thing left to say is that you can create your own Quix account if you want to try this for yourself.


Try Quix for yourself for free (no credit card, no time limit).

Start for free
Steve Rosam
words by
Steve Rosam, Full-stack developer

Steve Rosam is a Full-stack developer at Quix, where he creates and maintains solutions both in-house and for customers. Steve has worked as a software developer for two decades, previously in a variety of industries including automotive, finance, media and security.

Previous Post Next Post

Related content

View all
SQL CDC feature
Explainer | 9 Nov, 2022
Build a CDC pipeline with the Quix SQL Server connector
Create a CDC pipeline and publish data to Kafka topics in just a few minutes with our open source SQL Server connector.
Steve Rosam
words by
Steve Rosam, Full-stack developer
Blog 183 feature
Explainer | 23 Aug, 2022
Why industrial IoT is essential and how to implement it
The internet of things has expanded from small personal devices to warehouses and factories. This post will look at how IIoT impacts various industries and how to start or accelerate your transformation.
words by
Mike Rosam, CEO & Co-Founder
How to capture and store time series data
Explainer | 9 Aug, 2022
Four solutions for handling time series data
Most data in streaming applications such as IoT, finance, user behavior analysis and automotive is time-series data. Learn how to capture, process and apply it to get the most value from it.
Tomas Neubauer
words by
Tomáš Neubauer, CTO & Co-Founder

The Stream

Updates to your inbox

Get the data stream processing community's newsletter. It's for sharing insights, events and community-driven projects.

Background image