Semiotics of art reception: in between semiotic translation and synesthetic response

Humans -- and artists -- are capable of complex semiotic and experiential translations: we can translate reality, be it nature or culture, into an experience of sound, image, touch, smell, mime, emotion. We speak about 'semiotic' translation when we translate artistic work into another semiotic medium, when we translate the outer conditions of the experience -- e.g. translating a novel into a movie, a painting into a song, a performance into a narrative. But before experiencing semiotic translation, humans experience 'synesthetic' translation, moving beyond different possibilities of inner responses towards an artistic setting: for example by way of feeling, smelling, or hearing what can be seen. One holistic perception of artistic expression can be transposed into different modes, each one enriching the other. Both translations originate in the human possibilities of multimodal experience (e.g. blending theory of Turner and Fauconnier), cognitive fluidity (Mythen 1998) -- both theories referring to complex neurological responses.
Since long, semiotics has analyzed translation on the level of the sign or sign system, neglecting the (neurological) origins/ counterparts of these processes. This presentation will consider the importance of the human synesthetic possibilities and integrate these into a broader account of semiotic theory (Kress 1998, p76). It will analyse the complex experience of an artistic manifestation realized by way of an 'outer' and an 'inner' semiotic translation: at one hand an 'outer' complex comprehension which encompasses memory, cultural and aesthetic conventions, personal narratives, knowledge and expectations (Zeki & Lamb 1994), at the other hand the 'inner', rich synesthetic and multimodal cognitive and kinesthetic responses in body and brain acknowledged by recent neuropsychological and cognitive research (Spence 2001, Edelman & Tononi 2000, Thibault 2004).