A study on data processing optimization in smart manufacturing based on Spark SQL
Abstract
In smart manufacturing scenarios, the processing of massive high-dimensional time-series data often faces severe I/O amplification and deserialization overhead issues. Traditional relational databases, which rely on hardware-based data management models such as network storage and backend servers, suffer from long query execution times and insufficient scalability for cluster-wide queries and computational analysis, making it difficult to meet the demands of querying, indexing, and storing massive amounts of big data. To address this issue, this paper proposes a multi-level optimization scheme for underlying data streams based on Spark SQL. This scheme replaces traditional row-based storage with Parquet columnar storage and combines column pruning, predicate pushdown, and dynamic partition pruning techniques to reduce the amount of invalid data scanned during distributed queries. During the experimental phase, a bimodal data foundation was constructed. The results demonstrate that, within a cloud-native architecture, establishing physical partitions based on specific dimensions can effectively mitigate the high network latency caused by metadata lookups. Following optimization, response times for complex full-table aggregation queries can be significantly reduced. This research effectively alleviates underlying I/O pressure in high-dimensional industrial data analysis and provides a methodological reference for related big data engineering practices.