Skip to content
Open access

MMTNet: A Dual-Branch Multimodal Mutual-Transformer Network for Hyperspectral–LiDAR Land-Cover Classification

2026 · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · Vol 19, pp. 34556-34576 · 0 citations · 40 references

Abstract

Multimodal land-cover classification with hyperspectral imagery and light detection and ranging (LiDAR) data remains challenging because of spectral–spatial heterogeneity, modality differences, and limited local discriminability under small-sample supervision. This article proposes a dual-branch multimodal mutual-transformer network (MMTNet) composed of a dual-branch residual encoder (DB-Encoder), a mutual transformer, and a lightweight token-to-pixel (T2P) prediction head (T2P Head). The DB-Encoder extracts modality-specific spectral–spatial and structural features and maps the two branches to compatible token representations of the same dimension. The mutual transformer retains two dense, spatially indexed token streams and performs intramodal self-attention followed by simultaneous HSI-to-LiDAR and LiDAR-to-HSI full-sequence cross-attention. The T2P Head maps the fused local tokens to a spatial logits map and combines the center logits with a fixed local mean using a parameter-free coefficient <inline-formula><tex-math notation="LaTeX">$\alpha =0.10$</tex-math></inline-formula>. Experiments on Houston2013, Trento, and MUUFL use five fixed data splits and report mean <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> sample standard deviation. MMTNet achieves overall accuracy values of <inline-formula><tex-math notation="LaTeX">$\text{94.24}\pm \text{2.78}\%$</tex-math></inline-formula>, <inline-formula><tex-math notation="LaTeX">$\text{99.26}\pm \text{0.24}\%$</tex-math></inline-formula>, and <inline-formula><tex-math notation="LaTeX">$\text{85.62}\pm \text{1.71}\%$</tex-math></inline-formula>, respectively, demonstrating competitive and stable performance across the three benchmarks. Component, prediction-head, fusion-weight, and architectural sensitivity analyses further clarify the contribution and behavior of the proposed modules. An all-method standardized-input profile shows that MMTNet is performance-oriented rather than lightweight: its parameter and MAC counts are below the two heaviest baselines, while its measured latency lies near the middle of the comparison.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.