Virginia E. Strehle¹, Frank Tong¹; ¹Vanderbilt University

Deep neural networks (DNNs) trained for face recognition have surpassed human accuracy (Parde et al., 2023; Phillips et al., 2018). However, there is mixed evidence regarding how sensitive these models are to face shape (Abudarham et al., 2019; Strehle et al., 2024). Here, we asked if a face-trained DNN is truly sensitive to face shape and whether it resembles human perception by leveraging a 3D morphable model of face appearance (Paysan et al., 2009) to directly manipulate face shape and texture (Jozwik et al., 2022; Yildirim et al., 2020). We generated multiple trios of faces to be compared, which consisted of one target face and two alternate faces that differed in shape or texture to various degrees. Half of the trials presented the target and alternate faces from the front view, while the other half presented the alternates offset by +/-22.5° from the front view target. Human observers were asked to report which of the alternate faces was most similar to the target. For comparison, we extracted responses from the final fully connected layer of a face-trained Inception-ResNet-V1 network (Szegedy et al., 2017) and measured the Pearson correlational similarity between responses to each target face and its two alternates. Both humans and the DNN showed an overall shape bias in their face similarity judgments in the front view condition (Human: 70.1% of trials; DNN: 60.4% of trials) and also in the view-offset condition (Human: 60.07% of trials; DNN: 58.3% of trials). Moreover, human and DNN shape biases were positively related for both front-view and offset-view conditions (Pearson r = 0.51and r = 0.46, respectively). Our results indicate that humans exhibit an overall shape bias in their face similarity judgments, DNNs exhibit a similar bias, and more important, the face-trained DNN showed a significant correspondence with human judgments of face similarity.

Acknowledgements: This research was supported by NEI grants R01EY035157 to FT and P30EY008126 to the Vanderbilt Vision Research Center.