The proposed metamorphic testing technique can effectively reveal faults that would remain undetected with pure fuzzing, and is introduced as a novel metamorphic testing technique tailored to transpilers.
Abstract
Transpilers are increasingly used for software development, especially in industrial domains that rely on domain-specific languages (DSLs), to allow engineers to work with familiar concepts and appropriate abstractions. Ensuring the correctness of these instruments is therefore critical in many industrial settings. This paper observes that existing approaches for compiler testing hardly generalize to transpilers. Differential testing approaches are hindered as multiple equivalent implementations of the transpiler under test are seldom available in practice. The approaches based on metamorphic testing assume the ability to execute the compiled binaries, an assumption that cannot be always made for transpilers, which oftentimes produce results expressed as source code, requiring complex toolchains, hardware-in-the-loop setups, and depending on non trivial inputs. This paper introduces a novel metamorphic testing technique tailored to transpilers. Instead of reasoning about the runtime behavior of compiled programs, our approach defines metamorphic relations directly over the source code produced by the transpiler. These relations capture a property that we call mutation consistency of the (transpiled) programs: mutation-style changes in the input DSL program must induce predictable and structurally consistent changes in the generated output. We implemented this idea in a tool, MCP-Tester, and evaluated it through a case study conducted in the context of a technology-transfer project. Our current empirical results indicate that the proposed approach can effectively reveal faults that would remain undetected with pure fuzzing.
Across three COBOL-Java case studies, spanning two open-source programs and one internal production-like COBOL program and ranging from 430 to 4,114 source lines, Locksmith consistently improved coverage beyond input-search plateaus, reaching nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal production-like COBOL program.
Andras Ferenczi, Jordan Docherty, M.D. Bessonov et al.· arXiv.org· 0 citations
MongoDB maintains client libraries in a dozen programming languages, used by tens of thousands of organizations and millions of developers. Most are implemented natively rather than as wrappers around a shared core. Ensuring consistent behavior across these libraries, comprising millions of lines of code, is hard but essential. Over eleven years, we developed a specification-based testing approach: tests are written once in YAML and executed by language-specific interpreters for each library. We describe the evolution from many ad-hoc formats to a Unified Test Format, which allowed us to delete over 22,000 lines of test code. The rate of nonconformance bugs fell up to 86% in drivers that adopted YAML tests (though results varied). We report lessons learned about declarative test design, test architecture, schema evolution, and the limits of unification.
A. Jesse, A. Jesse Jiryu Davis, Jeremy Mikola et al.· 0 citations
Term rewriting systems are a common tool in automated reasoning and semantics of programming languages, and many practical applications require these systems to be convergent. While automated tools and theory exist to establish convergence, this paper is concerned with a practical method for testing it to quickly find useful counterexamples. Standard property-based testing approaches struggle here: exhaustively computing all normal forms is fundamentally flawed and too slow, while generating random normal forms makes counterexample minimization (shrinking) fragile due to dependencies on earlier generated test data. To solve this, we introduce a QuickCheck testing method based on generating and shrinking random execution traces. By checking if the first and last terms of a generated trace share the same deterministic normal form, we remove the data dependency between generators. This approach yields a property that efficiently finds counterexamples and enables fast, robust shrinking. We demonstrate the effectiveness of this method on various examples, ranging from group theory equations to distributed process calculus.
Koen Claessen· Proc. ACM Program. Lang.· 0 citations
WebAssembly (Wasm) has emerged as a widely adopted compilation target for systems programming languages, offering near-native performance and portability across platforms. However, the increasing use of code obfuscation techniques in Wasm binaries poses significant challenges for program analysis, malware inspection, and reverse engineering. In this paper, we investigate the capability of large language models (LLMs) to perform automated Wasm deobfuscation. We first provide a conceptual formulation of Wasm deobfuscation as a semantic recovery problem under unknown and potentially non-invertible program transformations, which naturally leads to an underdetermined reconstruction task. Building on this formulation, we introduce a three-tier evaluation hierarchy for assessing deobfuscation quality, consisting of syntax correctness, execution validity, and semantic similarity. We also employ fuzzing based validation as an intermediate evaluation procedure between execution validity and semantic similarity. By generating diverse input cases and comparing the resulting behaviors of the original and reconstructed programs, fuzzing enables us to assess whether semantic preservation holds beyond a limited set of manually specified test cases. We further interpret prompt design as an experimental configuration and study how it induces different inductive biases over the latent semantic hypothesis space explored by LLMs during reconstruction. To empirically study these factors, we construct a structured evaluation framework spanning multiple programming languages, obfuscation stages, and transformation regimes. Experiments with four state-of-the-art LLMs under multiple prompting strategies demonstrate that syntax-aware prompting significantly improves reconstruction accuracy; in particular, for the best performing LLM on C, applying the mono-shot prompting strategy under single obfuscation yields improvements of 33.06%, 7.43%, and 0.13 in syntax, execution, and semantic(LLM) performance, respectively, while virtualization-based obfuscation exposes an identifiability gap that limits semantic recovery without interpreter-level knowledge. These results provide empirical evidence and a conceptual framework for analyzing LLM-assisted deobfuscation as an inverse semantic reconstruction problem under adversarial program transformations.
