The evolution of data processing: why time to value is paramount

Last decade, companies wrestled with big data. The new challenge is how to handle data fast.

Data empowers business leaders to make decisions that increase revenue, enables people to automate machines and power products that can revolutionize every industry, from gaming to finance, mobility, telco and ecommerce.

“Data infrastructure is vital to the delivery of critical services and is required for the functioning of essential sectors of the economy, including financial systems, public utilities, industrial supply chains, media channels and telecommunications.”

“Digital revolution expanding infrastructure investment universe,” in WTW (formerly Willis Towers Watson)

The collection and use of data have expanded and evolved. Now there’s room to get even more value from your data, no matter where your company is in its data journey. In this post, I’ll talk a little about the evolution of data and then focus on how companies can get more from their data by reducing the time between data creation and its use.

The evolution of data from historical to real time

We’ve seen data trends evolve from the relational databases of the 90s through the explosion of solutions to manage big data over the past decade. The 2020s are about speed — enabling people to transform massive volumes of data into action faster.

Most data professionals see the value in fast or real-time data. Ninety percent of UK and US data professionals and IT decision-makers said they need to increase investment in real-time data analytics solutions in the near term, according to a survey by KX. But analytics is just the tip of the iceberg when it comes to processing data in real time.

Identifying use cases where faster data yields more value

Stream processing means more than simply building real-time dashboards. In most cases, overnight batch processing is sufficient for a dashboard that gets looked at a few times a week.

The real value comes from operationalizing data without going through the whole ETL (extract, transform and load) or ELT process. What’s more, adding context to data as it streams enables people across your organization to better access and use it.

Data loses value the longer it sits

The more time that passes between an event taking place and data about that event resurfacing in a usable form, the less valuable that data will be. Data provides the greatest value when it’s put to work immediately.

Use cases where latency makes data less valuable or irrelevant

I recently wrote about how reducing latency delivers a better customer experience, but speed is even more valuable in cases where a timely response is critical. Consider the difference fast data makes in these examples:

• Fraud detection: Detecting a suspicious spending pattern and freezing a credit card in real time prevents loss far better than spotting the same pattern the next day.
• User sentiment: Spotting a change in user sentiment in minutes instead of hours can make the difference between being in sync with your customers and sounding out of touch.
• Predictive or preventive action: Applications that monitor and react to IoT data in real time can prevent problems such as broken machinery in manufacturing and adverse events in the ICU.

“Data is a critical component of success for all fast-growth B2B companies.”

Dawn Capital

When development cycles get slowed down by data demands

With so many potential applications for data, more and more people within an organization want access to data to build workflows, products and better customer experiences. Companies collect data, but their teams often lack the expertise to clean, transform and properly store data for broader use. Even data scientists and engineers can struggle to wade through the massive amount of data that gets dumped into data lakes and warehouses.

Faced with data-related delays, Uber decided to make data more accessible without relying on a handful of data engineers.

Uber’s growing population of data analysts and city operations managers dedicated hours or even days to figure out, test and review code before putting it into production — and then had to wait for the next deployment cycle to release it. An internal study estimated that these delays cost them millions in lost productivity.

“For a fast-moving company like Uber, such delays hurt our business. An internal study estimated millions of dollars of productivity loss due to this.”

“No Code Workflow Orchestrator for Building Batch & Streaming Pipelines at Scale,” in the Uber Engineering Blog

Inspired by low-code/no-code platforms, Uber’s engineers designed a system that put the complexities of using real-time data in the background. By moving the complexity of processing data upstream, companies make data more accessible across the organization.

Like Uber’s engineering team (but without requiring you to hire legions of engineers), Quix manages the complexity of stream processing for organizations. Our vision is to empower companies of any size to benefit from a no-code/low-code interface.

What’s your next step to reduce your data’s time to value?

Companies already invest in data teams and infrastructure. More than 90% of companies participating in NewVantage Partners annual Data and AI Leadership Executive Survey said they are increasing investments in data and AI initiatives. And these companies see results. 92% of participating companies reported measurable business benefits this year, up from just 48% in 2017 and 70% in 2020.

We see a lot of companies moving from a database approach to data in motion. Companies progress from a limited consumption of a single data stream to unlocking the value of streaming data by ingesting more streams across the organization.

Here’s how we’re seeing leading companies progress on their journey from 2010s-era big data storage to 2020s-era stream processing:

• Streaming infrastructure: Connect siloed systems and teams with streaming infrastructure. At this stage, companies still write data to a database. It becomes someone else’s problem to figure out how data can be helpful down the road.
• Stream processing: Data teams use stream processing to gain intelligence from data as it moves from source to destination. This can start with small projects and expand throughout an organization.
• Expanding access to data: Unlock expertise in the organization by giving more people access to data when it’s at its most valuable. This can be done in a development environment that mirrors the company’s actual, real-time data — but the user’s sandbox is isolated so there is no possibility of breaking production environments.

How Quix helps teams transform data into products, faster

Getting data from the edge to a product is where Quix shines. We help small data teams build data pipelines more quickly and we help big teams skip over the complexity of building reliable stream processing architecture.

That means Quix helps your teams get to the fun stuff faster.

If you’d like to learn more about how Quix could help, book a consultation or join our Slack community to chat with us.