This paper proposes antenna selection-based beamforming optimization algorithms for full-duplex (FD) massive multiple-input multiple-output (mMIMO) large-scale arrays. The approach jointly selects the optimal transmit and receive antennas to optimize the propagation channels, thereby mitigating both self-interference (SI) and multi-user interference while preserving the desired signal strength. Illustrative results based on measured SI channels between crosspolarized 8 $\times$ 8 Tx and 8 $\times$ 8 Rx arrays demonstrate substantial improvements in both SI suppression and achievable total rate. On average, the proposed method achieves a 129.3% FD sumrate improvement over the Power-Greedy Selection scheme and a 65.5% improvement over the Random Selection scheme. Specifically, compared with the Power-Greedy and Random Selection schemes, the proposed method improves the uplink rate by average factors of 11.3 and 29.2, respectively, with maximum improvements of up to 13.9 and 35.3 times. An average 9.1 dB enhancement in SI suppression is observed. A comprehensive analysis of antenna selection effectiveness and downlink-uplink rate tradeoff in FD communications is provided.
This paper proposes a tri-hybrid beamforming (tri-HBF) scheme with antenna-selection (AS)-based reconfigurable sub-arrays for full-duplex (FD) massive multiple-input multiple-output (mMIMO) systems. A sub-connected HBF architecture is adopted, where AS is performed in a group-wise manner to avoid excessive switch-network and routing complexity. An alternating optimization (AO) algorithm is developed to jointly optimize the i) active antenna subsets considering a self-interference (SI)-aware utility, ii) analog beamformers through projected gradient ascent (PGA), iii) digital precoders/combiners via SI-aware regularized zero-forcing (RZF) and minimum mean-square error (MMSE) updates, and iv) DL/UL power allocation by successive convex approximation (SCA). To capture realistic electromagnetic coupling in FD mMIMO operation, experimental SI channels based on an 8x8 Tx-8x8 Rx FD array prototype are incorporated into the study. The proposed AS-aided tri-HBF optimization scheme exhibits robust convergence across various base station configurations and effectively balances desired-signal enhancement, SI mitigation, and multi-user interference suppression in FD mMIMO operation. Illustrative results show that selective activation can outperform full-array activation, achieving a 21.3% higher average sum-rate and a more consistent performance across user realizations, with power-efficiency benefits by reducing the active paths. A comprehensive study is conducted to characterize how the number of activated antennas affects the achievable rate, user-channel coherence, and SI suppression gain. Compared with various selection baselines, it achieves a 45.1% improvement in average sum-rate, with average DL and UL rate gains of 36.9% and 82.9%, respectively. In addition, beam-level isolation better than 63 dB is achieved, further confirming the effectiveness of the proposed SI-aware design.
