P138: Schema evolution at enterprise scale: Amazon Web Services study

Abstract

Schema evolution at enterprise scale: Amazon Web Services study Author: Sonu Kumar Singh (Senior Consultant — Cloud & AI Solutions Architecture, Capgemini US LLC) Professional Credential: Member, IEEE (Membership # 102728576) | ORCID: 0009-0002-9180-4946 Abstract The engineering problem behind schema evolution at enterprise scale is deceptively simple: teams want faster access to data and AI capabilities without surrendering correctness, control, or the ability to explain how a result was produced. This paper studies that problem across AWS. The emphasis is on the choices that matter in practice—data boundaries, metadata, identity, operational behavior, evidence, and the cost of coupling. To keep the discussion testable, schema evolution at enterprise scale is defined in operational terms. The paper focuses on the points where architecture choices become visible in behavior—how data is represented, how policies are enforced, how workloads fail and recover, and what evidence is left behind. That boundary is intentionally narrower than a feature survey and broad enough to capture the system-level trade-offs. The paper contributes a decision framework and a validation plan. It connects architecture to governance, reliability, cost, and measurable evidence, and it uses threat/risk modeling with operational validation as the primary research method. Where no experiment has been run, the paper says so directly and specifies what would have to be measured before an empirical conclusion could be defended. Architectural Research Scope Research Domain / Theme: Lakehouse & Analytics Architectural Scope: Amazon Web Services Core Research Question: How should schema evolution at enterprise scale be designed, governed, and empirically evaluated for AWS? Specification Standard: Full 20-page peer-level monograph featuring system topology diagrams, 7 empirical benchmark tables, and failure-mode analyses. Published as part of the Cloud, AI, and Distributed Data Systems: 500-Monograph Engineering Corpus.