As mentioned in my previous article, secondary analysis is all about “big data”—looking at the wider picture across different test campaigns. Validation teams need to consolidate vast volumes of test data into a central platform so they can compare tests across facilities, trace how a component behaved across years of test campaigns, or to connect a calibration change to a change in behaviour.

Ironically, European manufacturers were the pioneers of centralised, standardised test data management. The ASAM ODS standard defines an enterprise data model for measurement data. Audi began an ASAM ODS-based measurement-data-management project in 1999 and moved major parts of it into open source (openMDM, now an Eclipse project) by 2008. European manufacturers have therefore run structured, centralised test data systems for more than two decades.

If there was any weakness in this approach, it was the form it took. Each manufacturer built its own highly specific in-house platform, running on its own on-premise clusters, deeply tied to its own processes. These systems were built before cloud computing and before modern data engineering. They scale poorly, they’re difficult to integrate with modern analysis tools, and if you want to modernize, you’re faced with a long, labor-intensive migration project.

Those migrations are now happening though, mostly through cloud platform vendors. Major car brands such as Mercedes-Benz and BMW have modernized their measurement data analysis using cloud native products such Databricks and Azure Data Explorer. 

Nevertheless, the Chinese have been able to make this “cloud native” transition faster and earlier. 

Big data is part of China’s automotive DNA

To understand the Chinese aptitude for handling big data, it helps to zoom way out and look at national attitudes to data privacy and data collection. Generally speaking, Chinese consumers are more willing to sacrifice data privacy for more advanced technology. Surveys by McKinsey, KPMG and Deloitte between 2016 and 2023 consistently found Chinese consumers more willing than German or American consumers to share personal and vehicle data in return for connected services. 

Thus, there are fewer obstacles to centralized data collection, so more of it is collected from its 1.4 billion citizens each day. And all that data needs to be processed somewhere. Additionally, Beijing treats big data and artificial intelligence as core national priorities, offering direct policy support, subsidies, and coordinated regional pilot zones. Thus, there is a huge pool of knowledge, experience and talent to draw on when setting up infrastructure to process any kind of “big data”.

The newest generation of Chinese car manufacturers (for example, XPeng, NIO, and Li Auto), built deep cloud-native data platforms from the start, in close partnership with domestic cloud providers. These aren’t necessarily built to handle physical test data specifically, but they are sophisticated data processing projects nonetheless.

There is also a structural difference in how these platforms are organised. Chinese carmakers build them as horizontal, company-wide platforms owned by a central data team. Geely, for example, built an internal data platform called GDMP that carries research and development, production, supply, sales and service on one system (as described by Mirrorship, the vendor behind StarRocks). NIO built a similar company-wide platform, called DataSight. In Europe, the historical pattern was the opposite: each data system was built and owned by a specific department, so the infrastructure that ran an ASAM ODS server stayed inside the test department.

The Chinese aptitude for big data is also reflected in the influence of Chinese companies on the open-source ecosystem for data processing technologies.

China's open source strategy

Instead of competing with Silicon Valley directly, China is heavily investing in open source. This includes the big data stack. In fact, they’ve even taken over a key open source project that originally started in Europe. Namely, Apache Flink. 

Apache Flink began in 2010, as a research project at German universities including TU Berlin. The goal was to process very large datasets across many computers, much faster than the tools of the time. However, in 2019, Alibaba purchased the company behind Flink (DataArtisans, later renamed Ververica) heavily adapted its capabilities and built a whole ecosystem of companion open source projects around it (for example, Apache Paimon, a data lake that integrates with Apache Flink) 

Given the Chinese big data sophistication, does that mean they’re channeling it towards power data processing for automotive R&D? Not necessarily, there’s little public evidence online to prove this. However, they know they use sophisticated pipelines to process field telemetry and operational data (just like their European and US counterparts).

For example, Li Auto developed a cloud-native enterprise big data platform, or lakehouse architecture, to manage massive data inflows from electric vehicle telematics, manufacturing, and business operations.  It’s all based on open-source tools like Kafka, Flink, Spark, and JuiceFS on Kubernetes—a tech stack which enables them to diagnose vehicle faults in real time, store massive telemetry datasets, and run high-speed queries. 

Many of the components are open-source projects with roots in China (StarRocks is a spinoff from Apache Doris which was originally created by tech giant Baidu, and JuiceFS was founded by Chinese engineers now based in the US)

What's interesting is what Li Auto decided next. In late 2024, the team took the heaviest workload on this platform (the signal store for more than one million connected vehicles, each reporting around ten thousand signals) and moved it off the self-assembled stack onto Hologres, a managed real-time data warehouse from Alibaba Cloud. 

As they wrote in their earlier 2022 case study, managing their own stack was too complicated: every additional component the team assembled added another point of failure that the team itself had to operate. Hologres was still based on open source technologies, Li Auto just paid Alibaba to host and run for them. 

To remain competitive in a cut-throat domestic market, Chinese OEMs have had to innovate with their telemetry data processing pipelines (as well as on car software). Thus,  it’s not far-fetched to assume that some of that infrastructure is probably being channeled towards processing R&D data too, for use cases such as normalizing data from disparate sources or validating simulation models against physical test data.

What can European OEMs and vendors take away from these examples?

The most useful lesson from China is not that Chinese manufacturers have found a superior way to manage R&D test data. There is just too little public evidence for that. What the examples do show is that Chinese OEMs have become comfortable applying mainstream data-engineering architectures to automotive data at a very large scale.

Platforms at Li Auto, Geely and NIO use technologies and architectural patterns that would be familiar outside the automotive industry: distributed object storage, stream processing, lakehouse architectures, Kubernetes and managed cloud data services. Automotive data is treated less as a special category requiring its own infrastructure and more as another large, complex dataset requiring additional domain context.

That’s an important distinction for European automotive R&D teams to remember. Historically, measurement-data platforms combined automotive-specific functionality with specialised storage, indexing and processing infrastructure. Modern data platforms make it increasingly possible to separate those concerns. Generic infrastructure can handle storage, distributed processing and large-scale querying, while an automotive-specific layer handles signals, units, test metadata, provenance and engineering workflows.

Li Auto also provides a useful warning against building all the infrastructure internally. Its engineers initially assembled a sophisticated open-source platform themselves, but later moved one of its most demanding workloads to a managed Alibaba Cloud service because operating the underlying infrastructure had become too complex.

For European OEMs, this suggests that modernising secondary analysis does not necessarily require replacing decades of expertise in automotive data formats. The more useful question is which parts of the existing measurement-data stack genuinely need to remain automotive-specific, and which can now be delegated to mainstream data infrastructure.

For new European vendors, this creates an interesting product boundary. Established measurement-data products typically provide storage, processing and automotive domain functionality as one integrated system. A newer product can instead concentrate more heavily on making mainstream data infrastructure usable for automotive R&D: preserving engineering context, connecting measurements with test metadata, and making campaign-scale data accessible to engineers without requiring them to become data engineers.

The opportunity is therefore not simply to reproduce the Chinese platforms in Europe. Rather, the opportunity is to combine the automotive domain knowledge that European engineering organisations already have with data architectures that have developed much faster outside traditional automotive software.