Auditory, Visual, and Tactile Interactions During Active, Dynamic Touch

Even though extensive research on the multisensory integration of auditory and visual stimuli has been carried out, the mechanisms of integration of tactile stimuli with other senses remain less known and understood. Furthermore, the brain mechanisms associated with active and dynamic tactile exploration of a surface have not been extensively studied. In the context of the development of human-computer interaction (HCI) technologies, understanding these mechanisms remains of vital importance for the realization of multisensory devices and to improve the user experience of the general population, but also to benefit the use by clinical populations, e.g., people with visual impairments. The planned experiment aims to study multisensory integration during the active and dynamic tactile exploration of a surface (a natural texture or the screen of a multisensory tactile device). The primary hypotheses are that simultaneous auditory and/or visual stimulation during active tactile exploration of a surface will help participants form a mental representation of the shape or texture they are exploring, and that the recorded brain activity will be compatible with multisensory integration mechanisms at the level of the cerebral cortex. The planned project will include (1) behavioral (psychophysical) experiments, to assess participants' performance in discriminating the spatio-temporal location of tactile, auditory, visual, audio-tactile, audio-visual, and audio-visual-tactile on the screen of a multisensory tactile device and (2) surface electroencephalography (EEG) recording experiments, which will be employed to study the cortical mechanisms of multisensory integration during active and dynamic tactile exploration .