Abstract. Accurate camera pose estimation in urban environments remains challenging when reference imagery is generated from low-detailed, textureless 3D city models and must be matched against real world imagery. In this work we (i) extend our existing iterative object-basesd visual localization approach with an additional semantic feature and (ii) conduct a systematic baseline study of contemporary feature matching techniques on such cross-domain query-reference image pairs. As a first step to close the domain gap, we augment our iterative object-based visual localization pipeline with semantic masks derived from a pretrained semantic segmentation model. Intersection-over-Union between query and rendered masks is incorporated into the matching score, leading to a better pose accuracy. For the baseline study, we use a range of feature matching techniques: handcrafted (SIFT, AKAZE, ORB, FAST), learned detectors (XFeat, Key.Net, DeDoDe, DISK, AffNet), learned descriptors (XFeat, DISK, DeDoDe, HardNet), learned matchers (LightGlue, LoFTR), the line matcher SOLD2, and the learned matchers MINIMA-RoMa, MINIMA-LoFTR, MINIMA-XoFTR, and MatchAnything, which were trained on cross-modality datasets. The cross-modality focused matchers achieved the best results. For 20% 10%, 9%, and 7% of the evaluated query images the estimated camera pose had a translation error less than 5m and a rotation error less than 5◦. In this context, the other methods were only able to achieve a maximum success rate of 1.4%.
Yasmin Loeper, Markus Gerke, P. Fanta-Jende· The International Archives o...· 0 citations
Image matching is a core component of applications such as Simultaneous Localization and Mapping (SLAM), Visual Localization, and Structure from Motion (SfM). However, the local image features central to this task are vulnerable to inversion attacks, which enable adversaries to reconstruct privacy-sensitive scene content from local features. These attacks pose a particular threat in distributed computing scenarios where the pre-computed features leave edge devices to be processed by remote servers. In this work, we introduce Misanthrope, a novel privacy-preserving keypoint detector trained through self-distillation to avoid detecting keypoints on people---a predominant source of privacy-sensitive content in most localization scenarios---thus mitigating inversion attacks at the source rather than through post-hoc obfuscation. We demonstrate how inverted images from traditional feature detection pipelines can be used to detect and re-identify people in the scene, while Misanthrope is able to mitigate these attacks. Furthermore, Misanthrope maintains image matching performance on par with the state of the art and even surpasses it in challenging settings where people act as distractors, such as phototourism and in-the-wild odometry. On the Image Matching Challenge 2021 Phototourism test set, Misanthrope is the top-performing sparse feature extractor in 7 out of 9 scenes. We make our model and its evaluation script available here: https://github.com/fratopa/misanthrope
F. Vultaggio, Predrag Djindjic, Markus Gerke et al.· 0 citations