Enhanced multi-agent interaction modeling with differential transformers for trajectory prediction.
Predicting the future trajectories of multiple agents is intrinsically challenging due to complex dynamic agent-scene interactions, diverse motion patterns, and the inherent uncertainty of real-world behaviors. Transformer-based architectures have recently become a prevailing paradigm for multi-agent trajectory prediction, as they leverage the multi-head self-attention mechanism to flexibly aggregate information across agents and time steps into contextual representations, where each position is updated by taking a weighted sum over all input tokens. However, in the self-attention approach, it is unknown if the current input or the rest of the inputs has a stronger influence on the updated representation. Therefore, there exists information lost at each model layer and the accumulated information lost could have unpredictable consequences for trajectory prediction. To address this issue of information lost, this paper introduces a framework based on the differential Transformer architecture. It is a general framework that explicitly models the discrepancy between each position and its context to enhance multi-agent trajectory prediction. The core idea of our framework is to introduce a differential encoding layer that encodes the difference between a token's self-representation and the contextual representation aggregated from all other positions in the sequence. The learned differential encoding is combined with the original representation as the updated feature. By integrating differential encodings, the representational capability of the generated trajectory embeddings is improved. The implementation of our differential encoding module is highly parallelizable and can be efficiently integrated into the Transformer-based models. Extensive evaluations are conducted on two popular benchmark datasets. The experimental results show that our differential Transformer is a general architecture that consistently achieves improved performance compared to the base models.