Generative AI-Driven Data Lakehouse Evolution: Automated Schema Design and Adaptive Query Optimization

Abstract

The traditional architectures for managing the resultant enormous amounts of stored data have been found wanting in terms of supporting analytical workloads because heterogeneous data sources are growing at an exponential rate. In this work, we explore how data Lakehouse architecture through automated schema design and adaptive query optimization would be affected by the introductory of Generative Artificial Intelligence (GenAI). Here, we introduce an entirely new framework utilizing large language models (LLMs) and generative neural networks that infers, evolves and optimizes schema frameworks on its own based upon the change in incoming data ingestion patterns. It automatically adapts execution plans to achieve maximum performance by employing GenAI-trained schema synthesis approaches to build the most efficient execution plans with minimized latency and maximized throughput intermixed with Reinforcement Learning query planners tuned on past user behavior [5]. Experimental results show that schema design overhead can be significantly reduced, as well as query execution speed up and scalability on structured, semi-structured and unstructured data ecosystems. Second, this framework adopts a self-healing schema mechanism to visually discover semantic drift and automatically means of reconciling schema-level inconsistencies in their structure. It also argues for the inclusion of open access to generative intelligence directly within Lakehouse pipelines, creating a path toward fully autonomous self-optimizing data ecosystems. Live architecture Our research provides theoretical foundations and architectural patterns extended to be used in next generation intelligent data platforms.