Knowledge Accuracy and Response Characteristics of an On-Device Large Language Model in Building Environmental Engineering: a Preliminary Case Study of EXAONE 4.0 1.2B Using ChatGPT-4o as a Cloud Reference
Aug 2026· Architectural research· Vol 28· 0 citations· 29 references
Abstract
This preliminary case study examines EXAONE 4.0 1.2B, an on-device large language model (LLM), in building environmental engineering, using ChatGPT-4o as a high-capability cloud reference rather than a size-matched competitor. Ten Korean-language items were evaluated: four multiple-choice, three single-step calculations, and three short-answer questions covering indoor environmental quality, energy, and post-occupancy evaluation. The same simple prompt condition was applied to both models, with no model-specific prompt optimization; therefore, the results represent one standardized condition rather than each model’s maximum performance. Both models answered all seven objective items correctly, although EXAONE showed terminology confusion in its reasoning on a thermal-environment item. Five doctoral-level experts rated the short-answer responses for accuracy, completeness, and logical consistency. Mean scores were 4.18 for ChatGPT-4o and 2.98 for EXAONE. Fleiss’ kappa was 0.020 and 0.044, while free-marginal kappa was 0.333 and 0.222, respectively; the latter values indicate fair, not moderate, agreement, so absolute expert scores require cautious interpretation. Qualitatively, ChatGPT-4o more consistently organized responses around concepts and scope, whereas EXAONE tended to provide specific technical, operational, and Korean-regulatory content but occasionally omitted canonical elements or added unsupported detail. Because the item pool is small and only one on-device model was tested, the findings are exploratory and cannot be generalized to building environmental engineering as a whole or to on-device LLMs as a class. The results support further investigation of on-device LLMs as supervised offline assistants where connectivity or external data transmission is constrained, but not as unsupervised tools for engineering calculation or compliance decisions.
FLARE is proposed, a novel framework that endows VLAs with robust error recovery capabilities through a ``Retry" and ``Reset" Paradigm, and significantly improves task success and robustness.
Ganlong Zhao, Zijia Tang, Xingping Chen et al.· 3 citations
LifeSciBench is introduced, a benchmark of 750 expert-authored tasks designed to evaluate whether language models can handle realistic life science research work, with each constituent task paired with a human expert-written rubric.
Amelia Liu, Andrew Ho, Anne Marie Droste et al.· bioRxiv· 2 citations
Experimental results show that CritICL consistently outperforms standard in-context learning and achieves performance competitive with or superior to test-time scaling methods, while requiring significantly fewer generations and lower token cost.
Yu-Fan Wu, Yinghui He, Zhengyi Hu et al.· 1 citation
TestifAI, a deep learning testing framework for efficient and accurate estimation of robustness against combinations of perturbations, is proposed and partial model tomography is introduced, a novel approach to reconstructing model behaviour in a multi-perturbation space from tests that apply only a small number of perturbations.
Arooj Arif, T. Hartung, E. Botoeva et al.· 1 citation
The complex multi-energy coupling characteristics inherent to integrated energy system (IES) present unprecedented challenges for the implementation of low-carbon scheduling. Existing optimization methods often exhibit limitations in system scalability, algorithm adaptivity, and carbon reduction efficacy for complex IES. This paper proposes a Large Language Model (LLM)-Embedded Multi-Agent Reinforcement Learning (LEMARL) to address the aforementioned issues. The proposed method integrates the global perception capability of LLMs with the dynamic optimization capability of MARL. Specifically, the LLM-Embedded module generates high-quality reward functions and policy frameworks from a global perspective, while the MARL module leverages these LLM-generated strategies for distributed interactive iterations—greatly enhancing computation efficiency and scalability. Simulation results demonstrate that LEMARL reduces carbon emissions by 7.76% and simultaneously decreases operating costs by 4.49% in a small-scale IES. Furthermore, LEMARL also exhibits superior applicability and scalability in large-scale IES of the IEEE 141-bus power grid integrated with 51-node thermal system.
Chen Xia, Tong Gou, Yinliang Xu et al.· IEEE Transactions on Smart G...· 1 citation
General-purpose language models can reason and synthesize knowledge, but complex work also requires sustained interaction with files, information sources, and executable code, together with state maintenance, failure recovery, and verifiable delivery. We call this \emph{working capability}: sustained, verifiable progress toward a real-world objective. Apodex 1.1 develops this capability along two complementary dimensions. \emph{Environment Scaling} expands the diversity and verifiability of executable file, search, and code environments, while \emph{Agentic Coordination Scaling} trains agents to decompose long-horizon tasks, delegate parallel work, integrate asynchronous results, and replan. A shared execution harness and AgentOS maintain task state and provenance across tools and agents, and training turns environment trajectories and coordination traces into reliable behavior. Across complex professional work, finance, scientific research, mathematics, coding, and search, Apodex 1.1 reaches the leading performance band despite using a substantially smaller model than many frontier systems. The 35B-parameter Apodex 1.1 Mini further retains strong working capability in a locally deployable form. These results ground agentic intelligence in useful, verifiable work completed over time and advance our goal of building a \emph{Heavy-Duty Solver} for ambitious, long-running tasks.
Apodex Team B. An, B. Li, B. Wang et al.· 1 citation
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
