Touch in human mesh reconstruction

The human skin is designed to experience touch — the ability to perceive a stimulus that comes into contact with the body surface. The sense of touch is crucial because it allows us to experience physical sensations, create and deepen social relationships, and establish a connection with the world around us. The field of computer vision should be able to model and reconstruct the full human body surface and capture its complex interactions with ourselves, other humans, and the environment. Understanding contact through computer vision will enable advancements in virtual and augmented reality and impact other fields like robotics and behavioral science. Interestingly, many works in our field investigate interaction between humans and scenes, while research on self- and human-human contact remains relatively scarce in comparison.

Human beings engage in self-touch or self-contact multiple times a day, indicating its behavioral significance and prompting extensive research in behavioral- and neuroscience. For instance, studies have shown that facial self-touch is a recognized indicator of stress in adults.^[1] Infants frequently and spontaneously touch their own body, which serves as a way of exploration, facilitating the development of body awareness^[2] and even fetuses show increased self-contact when maternal stress is present.^[3] The patterns of self-touch are manifold as they vary in speed, trajectory, or movement duration^[4,5] and their exact function is still unknown. Some work argues that self-touch mainly serves for self-stabilization and self-calming^[5,6] or as a mechanism for down-regulation in high arousal states.^[6] Self-touch is also associated with emotional processes that interfere with working memory performance.^[7] In particular, suppressing self-touch among individuals who frequently touch their own body leads to significantly worse memory performance in haptic working memory tasks.^[8] Other research indicates a connection between different patterns of self-touch and neuropsychological state^[4,9] and mental arousal.^[10,11,12] In dialogues, self-touching colloquists are rated significantly more honest, outgoing, likable, and positively w.r.t. the working relationship compared to their non self-touching equivalent.^[13] This indicates a relevance of self-touch not only for self-regulation but also as an outwardly effective mechanism.

The relevance of interpersonal touch or human-human contact has also been investigated extensively in behavioral science, in particular the role of social touch.^[14] In fact, the body of research on social touch is extensive, and this paragraph can only offer a brief glimpse into interpersonal touch to highlight its relevance and functionality in human behavior. Beginning from early childhood, physical contact between parent and child establishes bonds and is associated with immediate stress reduction,^[15,16] enhanced object exploration,^[17] and long-term effects on behaviour.^[18,19,20] Early vocabulary items may consist of words often linked with caregiver touches.^[21] This is surprising because there is no reason why words like 'feet' should be spoken more often than 'diaper'. In older children, the avoidance of interpersonal touch can be a predictor of autism spectrum disorder.^[22,23] Social touch also has many effects in adulthood. Crusco and Wetzel^[24] show that a slight touch increased tips in restaurants, i.e. touch causes a more friendly behaviour towards the touch-giver. This effect, also known as the Midas touch, was replicated multiple times in subsequent research studies. For example, the exposure to social touch increases a bus driver's willingness to transport customers without having enough money for the ticket.^[25] Recent work found that, in virtual reality, agents with touch are perceived as more human-like.^[26] This confirms the importance of touch not only in real life but also in the virtual world.

Despite the great relevance of self- and human-human contact to learn about human behavior and states, most research on this topic is constrained by small group sizes because contact usually requires manual annotation as only a few rudimentary detection and reconstruction methods exist. This prevents understanding the importance and functionality of touch on human behavior at scale. The field of computer vision could advance the understanding of human social interactions by providing methods for 3D mesh reconstruction with accurate self- and mutual contact from images and video.

Unfortunately, self- and human-human contact has rarely been studied. One reason is that contact is rare in most human scan and motion capture (Mo. Cap.) datasets, because contact naturally leads to occlusion, which hampers data capturing. In body scan datasets, most poses avoid self-contact and in Mo. Cap. systems usually only a single person is captured. The implications for our field are evident: recent 3D motion generation methods can perfectly synthesize a single static person or human motion, but cannot generate two people shaking hands. Another reason why reconstructing accurate contact is neglected in 3D human pose and shape estimation is that most methods predominantly rely on 2D joint locations for supervision. However, 2D joints are not sufficient to accurately estimate the body surface, because one set of 2D joints can be explained by multiple body shapes and also by multiple poses when no ground-truth camera information is available. Priors, i.e. mathematical functions or models that incorporate prior knowledge about human pose and shape, are usually learned from scan and Mo. Cap. datasets that hardly contain contact poses. This leads to 3D mesh estimates that, when projected onto the image, satisfy reprojection constraints and may perfectly overlay with the image evidence. A rotation to the side, however, reveals that the estimated poses are not correct.

Being able to reconstruct and generate meshes with self- and mutual contact will facilitate the creation of avatars aligned with human behaviour which will let them appear more human-like, natural, and realistic.