Capability of multimodal large language models to describe gravitational wave images

Abstract

The advent of multi-messenger astronomy has highlighted the necessity for a comprehensive analysis of immense datasets generated by multiple astronomical observation instruments. The field of gravitational wave astronomy has employed deep learning for the efficient processing of data. However, the utilization of this technique has predominantly been confined to a single type of input data, such as images. While image-to-text applications with multimodal large language models (LLMs) are growing in other fields, a notable gap remains in the generation of textual descriptions of astrophysical and astronomical images. This study explores the potential of using multimodal LLMs to describe spectrogram images of anomalies captured by ground-based gravitational wave observatories. Three models, GPT-4, GPT-4o, and Gemini 1.5 Pro, were provided only with an image and a single prompt, without additional context. The generated text responses were then evaluated for length, meaning, and accuracy of the answers. All three models correctly identified each image as a spectrogram representation of gravitational wave data and generated appropriate explanations. However, they exhibited difficulties in detailed signal descriptions and source inference under context-free setting, indicating that further research is needed before practical application to gravitational wave data analysis. This study demonstrates the feasibility of using multimodal LLMs to transform spectrogram images into textual data and provides insight into their potential integration into gravitational wave data analysis workflows.