Model, decoder, and feature data for Cross-cue reconstruction of perceived 3D object structure from human visual cortex

Abstract

OverviewThis item contains model-related data used in the following paper:"Cross-cue reconstruction of perceived 3D object structure from human visual cortex." In preparation.The data include trained point-cloud autoencoder parameters, DNN latent features extracted from model-input point-cloud representations, and trained fMRI-to-feature decoders used in the reconstruction pipeline. AtlasNet is the primary autoencoder used for the main analyses, and a diffusion-based point-cloud autoencoder is included as a complementary model used for robustness checks.Source materials noteThis item distributes model parameters, DNN feature arrays, and fMRI-to-feature decoder parameters.Some files in this item were trained from, extracted from, or otherwise derived from point-cloud representations associated with third-party 3D object resources, including ShapeNet and 3D Warehouse resources used in the study. Accordingly, reuse of these files in relation to the underlying 3D object resources may be subject to the ShapeNet Terms of Use, the 3D Warehouse Terms of Use, and any applicable rights retained by the original model developers or other rights holders. Users are responsible for ensuring that their use of these files complies with all applicable terms and rights.The item-level license displayed by Figshare does not supersede or replace the terms and restrictions associated with the underlying third-party 3D model resources. It should not be interpreted as granting unrestricted reuse of these files for purposes that would require rights to the underlying third-party 3D object resources.Before using, redistributing, modifying, extracting, or otherwise reusing these files or their contents in relation to the underlying 3D object resources, please consult the applicable terms of use, including those of ShapeNet and 3D Warehouse.ContentsThe archives included in this item are as follows:Pretrained DNN model weightsdnn-weights-atlasnet.zip: trained AtlasNet autoencoder parameters.dnn-weights-diffusion-point-cloud-autoencoder.zip: trained diffusion-based point-cloud autoencoder parameters.True DNN featuresfeature-train-3d-natural-objects.zip: DNN latent features for the natural-object training set.feature-test-3d-natural-objects.zip: DNN latent features for the natural-object test set.feature-test-3d-artificial-objects.zip: DNN latent features for the artificial-object test set.Each archive includes features from both AtlasNet and the diffusion-based point-cloud autoencoder.Trained feature decodersdecoder-train-3d-natural-objects.zip: trained fMRI-to-feature decoders for the five subjects and visual ROIs.These decoders were trained on fMRI responses to 2D rendered images of the training natural objects and are used to predict DNN latent features from fMRI responses.Relationship to other items in the collectionThe model-input point-cloud representations used to extract the true DNN features are provided in the natural-object and artificial-object stimulus data items. The corresponding fMRI responses are provided in the fMRI response data item. The trained feature decoders in this item are used with those fMRI responses to generate decoded DNN features, which are then passed to the point-cloud autoencoder generators for 3D reconstruction. Reconstruction videos are provided in the natural-object and artificial-object reconstruction video items.CitationPlease cite the following when using this item:Figshare Collection: 10.6084/m9.figshare.c.8508462Associated paper: "Cross-cue reconstruction of perceived 3D object structure from human visual cortex." In preparation.ShapeNet: Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., & Yu, F. (2015). ShapeNet: An Information-Rich 3D Model Repository. arXiv. https://doi.org/10.48550/ARXIV.1512.030123D Warehouse: https://3dwarehouse.sketchup.com/AtlasNet: Groueix, T., Fisher, M., Kim, V. G., Russell, B. C., & Aubry, M. (2018). AtlasNet: A Papier-Mache Approach to Learning 3D Surface Generation. CVPR.Diffusion probabilistic model for 3D point clouds: Luo, S., & Hu, W. (2021). Diffusion Probabilistic Models for 3D Point Cloud Generation. CVPR.