Skip to content
Conference

A study on data processing optimization in smart manufacturing based on Spark SQL

Nov 2026 · International Conference on Algorithms, Image Processing, and Deep Learning (AIPDL 2026) · 0 citations

Abstract

In smart manufacturing scenarios, the processing of massive high-dimensional time-series data often faces severe I/O amplification and deserialization overhead issues. Traditional relational databases, which rely on hardware-based data management models such as network storage and backend servers, suffer from long query execution times and insufficient scalability for cluster-wide queries and computational analysis, making it difficult to meet the demands of querying, indexing, and storing massive amounts of big data. To address this issue, this paper proposes a multi-level optimization scheme for underlying data streams based on Spark SQL. This scheme replaces traditional row-based storage with Parquet columnar storage and combines column pruning, predicate pushdown, and dynamic partition pruning techniques to reduce the amount of invalid data scanned during distributed queries. During the experimental phase, a bimodal data foundation was constructed. The results demonstrate that, within a cloud-native architecture, establishing physical partitions based on specific dimensions can effectively mitigate the high network latency caused by metadata lookups. Following optimization, response times for complex full-table aggregation queries can be significantly reduced. This research effectively alleviates underlying I/O pressure in high-dimensional industrial data analysis and provides a methodological reference for related big data engineering practices.

View source

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.