Improving observability and reliability in SKA control software through TANGO-controls OpenTelemetry and CSP.LMC test analytics

Abstract

In a large-scale scientific project such as the Square Kilometre Array (SKA), reliability is critical. The ability to detect rare, hidden, and difficult-to-reproduce anomalies is essential and must be done as early as possible during development, well before integration and testing, commissioning, or production. In this context, robust Continuous Integration Continuous Delivery (CI/CD) practices and comprehensive software testing are mandatory requirements for all components. SKA Software Components rely on TANGO-Controls and run as containerised applications orchestrated via Kubernetes. Due to the complexity of the applications sometimes unpredictable behaviours can be observed and non-deterministic failures that must be carefully monitored and understood. To address these challenges, the Central Signal Processor Local Monitoring and Control software subsystem (CSP.LMC) has started a data-collection campaign aimed at gathering statistical information on test failures while also enabling systematic benchmarking of subsystem functionality. With the introduction of OpenTelemetry support in TANGO-Controls starting from version 10, new opportunities have emerged to significantly enhance observability within Tango Devices. This work investigates how the integration of Tango OpenTelemetry instrumentation with CSP.LMC’s test-result collection framework can provide deeper insight into software behaviour under anomalous conditions. Additionally, it explores how this combined approach can support more rigorous performance benchmarking and contribute to higher software quality and system reliability across the SKA ecosystem.