Abstract: In the 21st century, as our individual agency, cognition and subjectivity extends into, and through, our prosthetic digital technologies, the human subject is increasingly constituted as an assemblage of human-computer-communications networks. The physically intimate sensory experience of the human-machine cyborg body is most unmistakably evident in prosthetic implants and direct bodily attachments. However, in this paper I argue that the networked visual avatar body that we see in video games and online virtual environments creates a new type of “virtual cyborg body”. This assemblage represents a complex intermediation of the physical and the virtual — an exemplary instance of new symbiotic human-computer amalgamations that enable the intermingling of real and virtual bodies as well as of cognitive processes, subjectivities and identities. Psychoanalytical and phenomenological accounts of our prosthetic relationships with technology as well as neuroscientific studies of mirror neurons and autoscopic phenomena provide useful models to help us understand the lived experience of these intimate human-technology assemblages. The distribution of subjectivity, agency and affect between our online and offline bodies produces a “mixed reality” experience — an emerging paradigm for experience in the 21st century.
Our intimate relationships with new information and communication technologies have contributed to a profound change in the nature of human subjectivity and experience. In the 21st century, as our individual agency, cognition and subjectivity extends into, and through, our prosthetic digital technologies, the human subject is increasingly constituted as an assemblage of human-computer-communications networks.
In his seminal work Understanding Media, Marshall McLuhan famously describes new media forms as ‘technological extensions’ of man (1967). In this idea of technological prosthesis, the prosthetic is envisioned as an enhancement and extension of human faculties rather than as a replacement of a lost function. [1] McLuhan also describes media as “translators” and comments that: ‘[w]hat we call “mechanization” is a translation of nature, and of our own natures, into amplified and specialized forms’ (1967: 67).
Our prosthetic technological extensions enable us to amplify and extend ourselves in ways that profoundly affect the nature and scale of human communication and, therefore, of human consciousness and subjectivity. Unlike the earlier prosthetic technologies of the industrial revolution, which extended and replaced functions of the human body (e.g. industrial manufacturing processes and transportation technologies), increasingly our new information and communication technologies are coming to replace or extend functions of the human mind and psyche. The central nervous system is extended outside of the human body by means of communications prostheses, creating a technological extension of sensory perception, cognition and, in a sense, of consciousness itself.
In our use of new technologies, there is a blurring or loss of boundaries between the self and the environment. Our media technologies (television, radio, telephones, computers) allow us to extend our perceptual reach beyond our immediate physical environment. As McLuhan comments:
All media are extensions of some human faculty — psychic or physical. The book is an extension of the eye … Clothing, an extension of the skin … Electric circuitry, an extension of the central nervous system (McLuhan and Fiore 1967).
More recently, in an extension of McLuhan’s work, the term “psychotechnology” was coined to describe new technologies that extend or augment human sensory and cognitive functions (Kerckhove 1991a, 1991b; Fink 1999). Kerckhove describes psychotechnology as ‘any technical device which extends or emulates one human psychological feature or another, or a group of them’ (1991a: 267). As he explains:
Computer and video externalise many things we used to do internally, like thinking, remembering, calculating, designing, imagining, projecting, planning, creating and even, when applied to esthetic effects, feeling. For example, in virtual reality systems, it is possible to establish a biofeedback relationship between the computer and the user’s pulse, heartbeat, blood flow and skin conductivity. These emotional responses can be interpreted by the system and converted instantly into graphic and audio variations programmed into the VR environment. Thus psychotechnologies distribute outside the body critical physical, sensory, emotional and cognitive functions that emulate the human nervous system (1991a).
Phenomenological descriptions of how we experience our bodies and, through them, the world around us, also provide a useful framework within which we can understand how our technological prostheses (including the virtual avatar body) enable the physical human body and psyche to have an extended reach in physical and virtual environments.
According to Maurice Merleau-Ponty, it is through the interaction of the body in its surrounding environment that we come to understand both the world around us, and our own bodies/selves. As the body interacts with its environment, it generates a phenomenological experience of the world and of the body and, more importantly, of the relationship between them:
We grasp external space through our bodily situation. A “corporeal or postural schema” gives us at every moment a global, practical, and implicit notion of the relation between our bodies and things, of our hold on them. A system of possible movements, or “motor projects,” radiates from us to our environment. Our body is not in space like things; it inhabits or haunts space (Merleau-Ponty 1964: 5)
This phenomenological idea of the corporeal schema or body image is significant in that it is not restricted to the physical body itself. The spatiality of the phenomenological body image is not limited by the boundary of the skin, it is “extendible” through artifacts (Merleau-Ponty 1964). It is important to note here that the body image or schema is not just a visual representation of the body, but also includes the phenomenological experience of the body in action in its environment. This includes sensations of motility as well as sensory, kinaesthetic and proprioceptive perceptions and affects. The body schema is inherently malleable, expanding and contracting as it incorporates elements external to the body as prosthetic perceptual devices through which the individual senses and operates in the world around them:
The blind man’s stick has ceased to be an object for him, and is no longer perceived for itself; its point has become an area of sensitivity, extending the scope and active radius of touch, and providing a parallel to sight. …To get used to a hat, a car or a stick is to be transplanted into them, or conversely to incorporate them into the bulk of our own body. Habit expresses our power of dilating our being in the world, or changing our existence by appropriating fresh instruments (Merleau-Ponty 1962: 143).
Don Ihde expresses a similar idea in his discussion of “embodiment relations” [2] where ‘the experience of one’s body image is not fixed but malleably extendable and/or reducible in terms of the material or technological mediations that may be embodied’ (1979: 74). Our body image/schema expands to incorporate technological prostheses as we project sensory perceptions in and through them. Through the incorporation of prosthetic technologies, the plasticity and mutability of our body-image is readily apparent. As Ihde comments:
We are our bodies — but in that very basic notion one also discovers that our bodies have an amazing plasticity and polymorphism that is often brought out precisely in our relations with technologies. We are bodies in technologies (2002: 138).
The key techno-cultural figures of the cyborg and the avatar most strongly embody these ideas of technological extension and prosthesis demonstrating how individual agency, cognition and subjectivity can be extended into, and through, our prosthetic technologies.
The cyborg, a hybrid intermingling of flesh and machine, has become a key figure both in popular culture and in everyday reality. Short for ‘cybernetic organism,’ the word was originally coined by Manfred Clynes and Nathan Kline in an article about the need for self-regulating cybernetic human-machine systems to enable human space exploration (1960) but has since come to describe a much broader range of human-machine couplings, both metaphorical and literal.
The potentialities of the cyborg are explored in art, literature and film as well as in scientific and critical texts. As Donna Haraway points out in her Cyborg Manifesto, the cyborg is a figure of both the imagination and of reality:
By the late twentieth century, our time, a mythic time, we are all chimeras, theorized and fabricated hybrids of machine and organism; in short, we are cyborgs. The cyborg is our ontology; it gives us our politics. The cyborg is a condensed image of both imagination and material reality (1991: 150).
Although the figure of the cyborg represented in the media and popular culture typically involves a literal merging of human and machine, Haraway and other theorists (Hayles 1999; Gray 2001; Clark 2003) argue for a much broader definition of the cyborg. They champion a definition that includes our everyday interaction with, and dependence on, the myriad cybernetic technologies that make up our personal, social, economic, political and technological selves:
Cyborgs actually exist. About 10 percent of the current U.S. population are estimated to be cyborgs in the technical sense, including people with electric pacemakers, artificial joints, drug-implant systems, implanted corneal lenses, and artificial skin. A much higher percentage participates in occupations that make them into metaphoric cyborgs, including the computer keyboarder joined in a cybernetic circuit with the screen, the neurosurgeon guided by fiber-optic microscopy during an operation, and the adolescent game player in the local video-game arcade (Hayles 1999: 114-5).
Real life cyborgs exist side by side with their fictional counterparts constructing a complex cultural web of cross-coded signification. In ‘Split Subjects, Not Atoms; or How I Fell in Love With My Prosthesis’, Allucquere Rosanne Stone describes a lecture by the well-known physicist Stephen Hawking. Hawking, because of progressively debilitating amyotrophic lateral sclerosis, is severely paralysed and is unable to speak without the aid of a computer connected to a Votrax allophone generator (an artificial speech device) that he operates with the limited movement he still has left in his fingers. Talking about the experience of watching his lecture, Stone comments:
[T]here is Hawking. Sitting, as he always does, in his wheelchair, utterly motionless, except for his fingers on the joystick of the laptop; and on the floor to one side of him is the P.A. system microphone, nuzzling into the Votrax’s tiny loudspeaker.
And a thing happens in my head. Exactly where, I say to myself, is Hawking? … Who is it doing the talking up there on stage? In an important sense, Hawking doesn’t stop being Hawking at the edge of his visible body … a serious part of Hawking extends into the box on his lap. In mirror image, a serious part of that silicon and plastic assemblage extends into him as well (1994: 175).
It is clear that Hawkings’ subjectivity and agency is distributed throughout the machine-human assemblage Stone describes, fundamentally blurring the boundaries between the living and the non-living, the natural and the artificial, and the human and the machine.
And now, along with these physical cyborgs, we have the figure of the virtual cyborg — the avatar — a virtual prosthesis that can occupy the online domain of cyberspace. Virtual environment researchers Jeremy Bailenson and Jim Blascovich define an avatar as ‘a perceivable digital representation [in a virtual environment] whose behaviours are executed in real-time by a human being’ (2004: 64). As the Internet has become increasingly media-rich, graphical environments and graphical avatars have largely replaced the earlier text-based environments and identities of role-playing games such as MUDs (Multi-User Domains) and MOOs (MUDs Object Oriented). By assuming a graphical avatar, an individual is digitally embodied in a virtual environment and they can interact with other avatars and objects within that environment in real time. Research shows that using avatars as communicative proxies creates a strong sense of intersubjective presence and copresence in virtual environments (Schroeder 2002; Blascovich 2002; Taylor 2002).
Like the figure of the cyborg, the digital avatar is both a figure of social reality and of the imagination, represented in science fiction novels such as William Gibson’s Neuromancer (1984) and Neal Stephenson’s Snow Crash (1992), as well as in films like Brett Leonard’s The Lawnmower Man (1992), Barry Levinson’s Disclosure (1994) and the Wachowski Brothers’ The Matrix (1999).
The self-avatar assemblage represents a complex intermediation of the physical and the digital — an exemplary instance of new symbiotic human-computer amalgamations that enable the intermingling of real and virtual bodies as well as of cognitive processes, subjectivities and identities. Scott Bukatman terms the new construction of subjectivity and identity achieved through the human-computer assemblage as a ‘terminal identity’, ‘an unmistakably doubled articulation in which we find both the end of the subject and a new subjectivity constructed at the computer station or television screen’ (1993: 9). The high levels of psychological and emotional investment that people make in their avatars have also been extensively documented (Turkle 1995; Yee 2007; Suler n.d.; Cooper 2007).
With the digital avatar, the individual’s body is virtually re-embodied to enable the individual to enter the screen space. Subjectivity and affect are distributed throughout the self-avatar assemblage. As Vivian Sobchack comments:
All surface, electronic space cannot be inhabited by any body that is not also an electronic body. Such space both denies and prosthetically transforms the spectator’s physical human body so that subjectivity and affect free-float or free-fall or free-flow across a horizontal/vertical grid or, as is the case with all our electronic pocket communication devices, disappear into thin air. Subjectivity is at once decentred, dispersed, and completely extroverted — again erasing the modernist (and cinematic) dialectic between inside and outside and its synthesis of discontinuous time and discontiguous space in the coherence of conscious and embodied experience (2004: 159).
The interactive digital screen operates as a portal allowing the self-body to be virtually re-presented and re-embodied in the digital domain and so to simultaneously exist on both sides of the screen. As we inhabit the virtual body of the digital avatar we become virtual cyborgs entering into a human-machine assemblage. But, how does this new virtual body of the digital avatar operate in the virtual terrain on the other side of the screen, and what is its relationship to the physical body “left behind?”
Early narratives of cyberspace and virtual reality made much of the supposedly disembodied nature of the experience, with the physical body being “left behind” as the disembodied mind or consciousness entered the virtual domain. As John Perry Barlow famously commented: ‘Nothing could be more disembodied or insensate than the experience of cyberspace. It’s like having your everything amputated’ (Barlow 2000). In Barlow’s description of his early experience of VR (using a head mounted display and dataglove), his “image body” or agency in the virtual terrain was represented by a disembodied floating hand (cybernetically connected to his physical body via the dataglove):
Suddenly I don’t have a body anymore. All that remains of the aging shambles which usually constitutes my corporeal self is a glowing, golden hand floating before me like Macbeth’s dagger …In this pulsating new landscape, I’ve been reduced to a point of view (Barlow 2000).
Barlow experiences a strong feeling of disembodiment as his physical body is left behind: ‘… I know where I left my body. It’s in a room called Cyberia in a building called Autodesk in a town called Sausalito, California.’ Melinda Rackham, who also quotes Barlow’s “amputation” experience of VR, goes on to compare the experience of VR to a state of quadriplegia or of an anaesthetised but still mentally active patient (2004: 65). However, this sense of disembodiment and being reduced to a visual “point of view” is clearly not the whole story of the virtual experience.
Ingrid Richardson argues that rather than the body disappearing in virtual reality, ‘an altered technosoma — a cybersoma — emerges at the interface’ (2003: 142). Richardson describes the VR-body as a ‘technosomatic intercorporeality’ (2003:153) and comments that:
Contesting notions of VR as a disembodying medium thus requires a shift from thinking of the virtual as de-corporealised subjectivity, toward a notion of embodiment as incorporating the virtual, as a way of having/being another kind of body (2003: 149).
Mark Hansen makes a similar point in his book Bodies in Code (2006) arguing that the ‘body-in-code’ we experience through our technological prostheses is in fact a technical mediation of the body schema — ‘a body whose embodiment is realized, and can only be realized, in conjunction with technics’ (2006: 20). For Hansen, this technical mediation of the human body is part of the ongoing evolution and ‘technogenesis of the human’ (2006: 21).
The use of the first person perspective in video games and virtual worlds can create a strongly immersive and immediate experience in which virtual actions and experiences are kinaesthetically transferred to the offline physical body. As neuroscientist H. Henrik Ehrsson comments: ‘The first-person visual perspective is critically important for the in-body experience… In other words, we feel that our self is located where the eyes are’ (Ehrsson cited in Madrigal 2008). In a review for Wired magazine, Clive Thompson describes his experience of Mirror’s Edge, a first-person game that triggers a strong sense of the body’s proprioception:
When you run, you see your hands pumping up and down in front of you. When you jump, your feet briefly jut up into eyeshot — precisely as they do when you’re vaulting over a hurdle in real life. And when you tuck down into a somersault, you’re looking at your thighs as the world spins around you…Only 15 minutes into the game, my mouth began overproducing saliva, and I had to pause the action for a few seconds to avoid carsickness (Thompson 2008).
So, in fact, our new avatar forms need not be seen as disembodied virtual entities where we leave the corporeal “meat” body behind. Instead, they are complex new expressions of prosthetic re-embodiment through which our physical bodies and subjectivities extend themselves into the virtual terrain. Indeed, with the emergence of the digital avatar, narratives of disembodied subjectivities or consciousnesses roaming through cyberspace have largely been replaced by a renewed interest in the body and an awareness of the importance of embodiment in virtual spaces. This is particularly the case as we shift from the first person perspective of VR that Barlow describes to the more commonly used third person perspective where the individual can see their virtually embodied avatar self.
Virtual re-embodiment in the form of the digital avatar plays an important role in creating a strong sense of presence and co-presence in virtual environments. While the first person perspective creates a high sense of immersion in the virtual game world, the third person perspective may create a higher sense of psychological and physical identification with the player’s avatar body. In ‘Living Digitally: Embodiment in Virtual Worlds’ T.L. Taylor comments: ‘Users do not simply roam through the space as “mind,” but find themselves grounded in the practice of the body, and thus in the world’ (2002: 42). An individual’s presence is signified by the visual presence of their avatar:
In multi-user worlds it is not just through the inclusion of a representation of self that presence is built. It is instead through the use of a body as material in the dynamic performance of identity and social life that users come to be “made real” — that they come to experience immersion (Taylor 2002: 42)
Watching spectator-participants navigate their avatar bodies through virtual environments, we witness a virtual re-embodiment that does not simplistically leave the physical body behind at the computer keyboard. It brings the socio-cultural signification of the physical body, as well as Its sensations and affect, along for the ride. This interplay of the physical human body (along with its socio-cultural meanings) and the prosthetic avatar body constitutes a complex new form of distributed embodiment and agency.
This is particularly true in the case of dancers, actors and performance artists where the importance of the physical body and the intensely felt mind-body connection means that their interest in virtual embodiment is grounded in the experience of the physical body. Rather than the virtual body being the primary focus, it is the connections and interaction between the physical and the virtual that become productive sites of performative experimentation. The work of actors, dancers and performers utilising video doubles, avatars and virtual reality technologies provides some exemplary cases of the phenomenological experience of integrated feedback and connection between physical and virtual bodies. As Steve Dixon comments, in dance and performance, the virtual body ‘operates as an index, as another trace and representation of the always already physical body’ (2007: 215) rather than representing an immaterial disembodied escape from the physical body. Performers experience a splitting of subjectivity and sensation when they enter virtual environments as they simultaneously experience their physical body (experienced in first person), and their digitally re-embodied avatar body (typically experienced as a visual image from a third person perspective). Awareness, consciousness and subjectivity oscillate between these different subjective and spatial locations.
In his virtual dance performance collaboration with Diane Gromala (Dancing with the Virtual Dervish: Virtual Bodies), Yacov Sharir performed within a gigantic virtual representation of Gromala’s body which he navigated through using a head-mounted display (HMD) and dataglove. Video images of his physical body were projected into the virtual body; for Sharir, this created a strange sense of being doubly embodied (in his physical body and his digitised video double). In ‘Virtually Dancing’ (n.d.) Sharir comments on the feelings of immersion and anxiety he experienced during this performance:
When I experience the entrance into a computerized simulated virtual world, I am able to reference or “see” my digitized body within the simulation. Simultaneously, I sense my existence in the physical world. As I target my vision and/or move my hand forward, I am able to navigate through the simulation-birdlike. As my perception accommodates itself to a 3-D illusion, I experience a sense of being in another, additional skin — I feel immersed. At the same time, I have this sense of heightened anxiety, caused by the doubling of my own body image. The sensation of disembodiment cannot be disconnected from the sensation of embodiment; that is, I feel the physicality, the groundedness of gravity simultaneously with the sense of immersion and altered abilities, such as the ability to “fly” through the simulation (Sharir n.d.).
One way that we can theorise the relationship between the physical body and the virtual body of the avatar is by looking at it as a shift between the embodied first person perspective and the disembodied third person viewpoint. That is, between our experience of being in our physical body and looking at the world and that where we see and experience ourselves as a semi-autonomous prosthetic image. This oscillation between the experience of the first person and the third person is explicit in many video games and virtual worlds where you can switch your camera view (P.O.V) between first person and third person views.
These different experiential perspectives are encapsulated in Ihde’s description of the embodied “here body” of the physical body (which he links with Maurice Merleau-Ponty’s “lived body”, the corps vécu). Ihde compares the embodied ‘here body’ with the disembodied ‘there body’ of the virtual image-body (2002: 6). The phenomenological experience of the “there body” (where the self sees and experiences itself as a visible other) can be read as a shift from the first person perspective of the embodied phenomenological body (self), to a third person perspective where the individual experiences their body as an external image (other).
This experience of seeing oneself as an image initiates a profound split in subjectivity and experience of the self. Simultaneously feeling ourselves in our bodies as well as seeing ourselves exteriorised as an image generates an oscillation between the experience of self as self and self as other.
This switch from the phenomenologically-embodied first person perspective, located within the body looking out at the world, to the third person perspective of seeing oneself from the outside originates with the mirror image. This is understood as both the literal mirror image as discussed by Jacques Lacan, and also John Horton Cooley’s idea of the “looking glass self” — the imagined image of the self as seen by the other.
The third person view of the avatar body can also be theorised as a type of out-of-body experience (OBE) where the individual sees himself as if from a location outside of his physical body. Australian new media artist Adam Nash describes this disembodied out-of-body experience in VNet (an open source multi-user virtual environment), where the individual’s point of view is dissociated from that of their avatar, leading to a strange sense of ghostly disembodiment:
The way it works currently, the OOBE view (Out Of Body Experience, i.e., objective view of one’s own avatar) allows you to move independently of your avatar’s geometry, which raises the question that if the position from which you are viewing the world is a different position from your avatar, is your avatar still representing you? If yes, then it shows that the idea that the 3D space is an actual space is notional at best. If no, then have you died and are now a virtual ghost? (Nash 2003).
Although Nash describes the out-of-body experience as one of disembodiment, I would argue that what we are witnessing here is something altogether more strange and complex. What is actually happening in the self-avatar assemblage is a dual embodiment rather than a disembodiment. To understand this phenomenon more fully, it is instructive to look more closely at a wider range of out-of-body (OBE) experiences.
Clinically, the OBE is associated with the phenomenon of autoscopic hallucination (literally “seeing oneself”) where the individual has the experience of seeing himself in extrapersonal space (Blanke et al. 2004). What distinguishes these two different phenomena is a shift in the point of view from which the phenomenological experience of seeing originates. In the OBE, the individual seems to see himself from a position outside of his physical body whereas in the autoscopic hallucination the individual remains phenomenologically located inside of his physical body (“here body”) while having the experience of looking at an externalised doppelgänger image of himself (the “there body”).
Neurologist Peter Brugger (2002) classifies both autoscopic hallucinations and OBEs as types of autoscopic phenomena along with an additional transitive or intermediate experience called heautoscopy where the point of view oscillates between being located in the physical body and in the parasomatic (or virtual) body.[3] Accounts of out-of-body experiences and autoscopic phenomena are frequent in folklore, mythology and accounts of spiritual experiences.They have also attracted the interest of neurologists and psychiatrists, who have observed these phenomena in patients suffering from neurological diseases such as epilepsy and migraines as well as psychiatric conditions such as schizophrenia, depression and dissociative disorders (Blanke et al. 2004).
However, what is fascinating, in the context of our discussion of the self-avatar assemblage, is the way these autoscopic phenomena appear to be re-created in virtual environments by the individual’s simultaneous (or oscillating) experience of the physical “here body” and the virtual avatar “there body”.
This split subjectivity is reinforced in some video games and virtual worlds by the ability to switch views between first person, where you look out from behind your avatar’s eyes, and third person, where you can see your avatar in the virtual environment. When the avatar is seen from a third person point of view, this experience of being both self and other is intensified as the individual simultaneously projects her agency into the virtual body of the avatar, but also maintains a third person perspective watching her avatar perform and interact with other avatars.
As we have seen, Ihde distinguishes between the physical “here body” and the virtual image “there body”, but is it possible that the virtual body may also be experienced as a “here body”?
Interestingly, recent scientific research and experiments have confirmed that bodily self-consciousness can indeed be spatially displaced outside the boundaries of the physical body and into bodily prostheses and virtual bodies. In the so-called “rubber hand illusion” (RHI), synchronous stroking of a (seen) fake rubber hand and the participant’s (unseen) physical hand results in the participant attributing the sensation they feel to the stimulation of the fake hand and feeling that the fake hand is part of their own body (Lenggenhager et al 2007). Where there is a multi-sensory conflict, vision typically takes precedence over proprioception and touch, resulting in physical sensation being cognitively remapped and experienced outside of the body. This phenomenon, called “proprioceptive drift”, is also experienced with spatially displaced virtual body images. Experiments with whole-body virtual images suggest that the spatial unity between the self and the body can be disrupted so that “selfhood” itself can be subjectively experienced outside the boundaries of the physical body (Lenggenhager et al 2007; Ehrsson 2007, Slater et al 2008). Researchers have created an out-of-body experience by using virtual reality goggles to show participants virtual images of their own bodies. The sight of their spatially displaced virtual bodies being touched, combined with the experience of their real bodies being touched, created a sense in the participants that they had moved outside of their physical bodies and into their virtual body.
Virtual reality systems that embody tactile or other kinaesthetic feedback intensify this sense of out-of-body experience by transferring individuals’ sensory-perceptual apparatus from their physical bodies to their virtual bodies. Early VR systems typically incorporated a head-mounted display (HMD) and a dataglove that relocated touch into the prosthetic virtual hand that appeared in the immersive VR environment. Full body datasuits with pressure sensors and activators can also enable virtual sensations to be felt by the physical body. Motion capture systems can be used to map movement from the physical to the virtual body, leading to complex sensory-perceptual feedback loops that transfer and distribute sensations between physical and virtual bodies.
In Stahl Stenslie’s CyberSM (1993) and inter_skin (1993), haptic bodysuits were worn by participants so that physical touch and sensation could literally be transferred from one person to another. The sensation of one participant stroking her own breast was transferred so that it is “felt” by the remotely connected participant. Dubbed ‘one of the fathers of cybersex’ (Popper 2007: 258), Stenslie’s haptic bodysuits excited many with the titillating possibility of turning autoeroticism into participatory virtual sex.
However, these fully immersive VR systems are still a rarity. The most common experience of virtual reality is so-called “desktop VR”, which is the ubiquitous experience of both video games and virtual worlds. However, even in these low-end VR environments, where there is no direct tactile or kinaesthetic sensory feedback, sensations experienced outside of the physical body can be cognitively remapped and “felt” by the physical body. While a visually triggered physical sensation is clearly not the same thing as an actual physical touch, it would appear that “seeing” a virtual movement or touch can be enough to remap and re-create that experience so that it can be, in some sense, felt or mirrored by the individual’s physical body. Here, the visual sense (and, to a lesser extent, sound) becomes a synaesthetic stand-in for the full body sensorium. Tactility, kinaesthesia and proprioception are mapped using vision and translated back into the body. A physical experience of a visually triggered sensation of touch has been described by participants of Paul Sermon’s telematic artworks such as Telematic Dreaming (1993) where video avatars are chroma keyed together so that they can virtually touch each other via a shared third composite image (Sermon 2004). The dancer Susan Kozel, who performed in Telematic Dreaming over a four week period, described feeling “little electric shocks pass through [her] body” as a visitor to the installation caressed her virtual video body. On another occasion she physically doubled over when a man violently elbowed her video image in the stomach, surprised at her reaction because her physical body hadn’t actually been attacked, nevertheless she “felt something” (Kozel 1998).
Mark Hansen also highlights what he terms the ‘primordial tactility’ of the phenomenological human body in its interactions with virtual spaces (2006). In this context Hansen draws on Shaun Gallagher’s interpretation of Merleau-Ponty’s work, which distinguishes between the body image as a visual representation of the body and the body schema as the spatially and tactilely felt experience of the body within its environment (Hansen 2006). Hansen gives primacy to the notion of the body schema and the tactile nature of embodied experience. However, while Hansen’s work acts as an important corrective to the ocularcentric focus of the vast majority of writing about virtual reality and cyberspace, his own writing has the opposite tendency of failing to fully acknowledge the importance of the visual image as a source of sensory perception and feedback. I would argue that it is the interplay of vision (and the image) with our other senses, including the aural, tactile and kinaesthetic senses, that creates such a strong sense of immersion in virtual environments.
Indeed it is the complex imbrication of touch and vision or, as Hansen describes it, the ‘transductive correlation of vision and touch’ (2006: 82) that enables touch to be ‘extended beyond the boundary of the skin’ (79) by the exteriorisation of vision. The connected experience of our physical “here bodies” and our visually imaged “there bodies” creates a complex mixed reality experience where subjectivity, sensation and affect are distributed from the virtual to the physical in the self-avatar assemblage.
The burgeoning field of neuroscientific research into mirror neurons helps to explain this phenomenon. Experiments show that areas of the brain collectively known as the “mirror neuron system” respond not only when individuals perform an action themselves but also when they watch someone else perform that action. While it is not the same thing as first hand physical experience, watching someone pick up an object triggers a similar brain response to actually picking up the object yourself. The activation of mirror neurons and corresponding physical motor responses are also triggered by visual images, pornography being a key example (Ponseti et al 2006; Mouras et al 2008).Watching someone cry, being hit, or expressing emotion, can also trigger empathetic mirror neuron responses in the watching individual (Ramachandran 2001; Gallese 2003; Rizzolatti and Craighero 2004; Jabbi et al. 2006).
Vittorio Gallese sees the phenomenon of mirror neurons as the underlying factor in creating an awareness of the subjectivity of others and in creating intersubjective empathy — he describes this as a shared ‘manifold of intersubjectivity’ (2003: 172). The role of mirror neurons has also been linked with human mimicry, theory of mind, learning and language acquisition (Ramachandran 2001).
While much of the current research focuses on how mirror neurons are triggered in response to the actions of others, it is interesting to theorise about what happens when the other that is seen and empathised with is a spatially displaced virtual body image of the self. Ramachandran speculates that mirror neurons may play a role in self-reflection and introspection:
…when you introspect you have a sense of yourself watching yourself from above; I’m doing things and I’m watching myself doing things. It’s obvious that mirror neurons might be involved there because just as you’re imagining the other person’s point of view looking at a peanut, you can imagine the other person’s point of view looking at yourself (Ramachandran 2007).
The reaction of our physical bodies to our virtual screen images reveals complex synaesthetic cross-modal transfers of sensory feedback. Through the virtual prosthesis of the avatar body, the individual can sense and explore the virtual realm and these experiences can be transferred back to and in some sense “felt” by the offline self. What happens to our virtual bodies triggers empathetic kinaesthetic experiences and feelings in our physical bodies as affect and sensation are distributed throughout the mixed reality complex of our physical and virtual selves.
The connection between self and avatar is built, developed and strengthened in an on-going process. When we first enter a virtual world, we have to learn how to operate our avatar body. It is with the avatar’s body that we explore and experience different virtual environments in an analogous way to how we walk around or drive in the physical world.
In Synthetic Worlds (2005), Edward Castronova explicitly compares the virtual avatar body to a car which user drive around in to experience virtual environments:
When we visit a virtual world, we do so by inhabiting a body that exists there, and only there. The virtual body, like the Earth body, is an avatar. When visiting a virtual world, one treats the avatar in that world like a vehicle of the self, a car that your mind is driving. You “get in,” look out the window through your virtual eyes, and then drive around by making your virtual body move. The avatar mediates our self in the virtual world: we inhabit it; we drive it; we receive all of our sensory information about the world from its standpoint (2005: 5).
As participants develop and internalise the appropriate skills to operate their avatars, the gap between the individual and their avatar decreases. Their identification with the avatar intensifies as movement is delegated from the physical to the virtual body. Like learning how to drive a car or ride a bike, once the skills are learned and mastered they become second nature and functionally invisible and the individual can then feel fully immersed in the activity. This experience is analogous to Mihaly Csikszentmihalyi’s concept of “flow” where the individual feels fully immersed in an activity (1975; 1991). [4] Flow occurs in a zone between boredom (where the task is too easy) and stress (where the task is too challenging). Many of the same factors that create a sense of flow are also associated with the way individuals operate and identify with their avatars. If the interface for controlling the avatar is too difficult or other technological impediments intrude, flow is disrupted and the participants’ sense of identification and connection is likewise disrupted. However, when the interface is mastered and flow occurs, individuals’ physical manipulation and operation of their digital avatars in real-time creates a strong sense of connection and identification.
As Castronova suggests, learning how to operate your avatar is a bit like learning how to drive, you have to learn what keys and controls to use to operate your new virtual body. Although the movements of the human controller and the digital avatar are synchronised and controlled in real-time (or with a brief lag), the movements of the digital avatar body typically do not correspond directly with those of the physical body. In the majority of today’s virtual worlds and video games, simple keyboard inputs and movements of the mouse control avatars’ actions, movements and speech. Arrow keys move your avatar around and you generally talk to other avatars via cartoon-like speech bubbles or dialogue-box windows. In some cases text-to-speech synthesis programs or live audio streaming are used so the avatar can speak out loud.
Learning to control our new digital avatars can take some time and can be uncomfortably counter-intuitive at first. When there is a disparity between the motivation and intention of the human controller and the actual actions and movement of the avatar on screen, there lies the potential for a great deal of frustration and feelings of uncanniness and alienation.
When you enter a virtual world like Second Life for the first time, you are a “newbie” and, like a newborn infant, you have to learn how to move and communicate all over again. As a newbie, you can feel a bit like the uncoordinated infant that Lacan describes in the mirror stage, struggling to control a body that is experienced as incoherent and out of control. However, unlike Lacan’s infant, who sees its reflected mirror image as reassuringly coherent and unified, in this new situation the situation is reversed. While we experience our physical bodies as coherent and under our control, the digital avatar that we see reflected on the screen is not one that is gratifyingly unified and coherent, but one that is uncoordinated and uncontrollable, or at least it is until we learn how to operate it.
In this new virtual environment, if you want to move your virtual limbs to walk around, you have to use a mouse or joystick or arrows keys on your computer keyboard. To change your style of walking or initiate another movement such as dancing or sitting, you have to select these various options via onscreen menus. Unlike more sophisticated motion capture technologies, where the movements of the physical body are transferred to the virtual avatar body in a much more natural and intuitive fashion, this interface is in no way natural or transparent. It needs to be learned and internalised before you can start to more seamlessly identify with your onscreen avatar. Until then, your avatar feels more like an unruly and uncontrollable puppet whose actions are frequently unpredictable and unintended. [5]
In Second Life, it is common to experience moments of frustration and uncanniness when your avatar just won’t do what you want it to. This can be the result of inexperience and lack of skills in negotiating the world or as a result of technical problems (for example, a slow graphics card or a congested network may cause lag in your avatar’s movements, or your avatar may appear in a new location without its clothes because they haven’t downloaded yet). Software presets that control various aspects of the way your avatar moves, behaves and looks (for example, the jerky default avatar walk and the Americanised voice that accompanies default animations in Second Life), also can add to feelings of alienation and frustration.
In my first forays into Second Life, it was common for me to walk my avatar into walls or into water and for it to perform strange, jerky movements seemingly beyond my volition. For no apparent reason my avatar’s arm would suddenly twist strangely behind its back and its head would move from side to side. (I realised later this happened because I was “pointing” at something behind my avatar’s body. When you click on another avatar or an object of interest, your avatar’s hand “points” in that direction so if that object is behind or above your avatar, it will have to contort to point in the appropriate direction.) On other occasions I would try to walk or turn around, but due to lags or faulty keyboard commands, my avatar body just would not respond the way I wanted it to.
Faster system responses and controller free interfaces using cameras to map users’ movements onto their virtual avatars may go a long way in helping to overcome some of these issues. In Japan, OKI Electric Industry is developing FaceCommunicator® [6], a proprietary software application that enables users to control the movement and expressions of their avatar by using a camera to detect the movement of the user’s eyes and eyebrows, and then using those inputs to generate synchronised animated movements in the avatar.
New games interfaces such as Microsoft’s controller free Project Natal system for Xbox 360 uses voice recognition technology and a camera to capture and map an individual’s physical movements onto the virtual avatar body. Lev Grossman, a journalist for Time magazine, describes his experience of using the Project Natal system:
It’s weird to be playing a game with nothing in your hands — if you’ve ever played a theremin, the sensation of playing with Project Natal is not dissimilar. It’s spooky. But it’s also very immersive. When a ball comes bounding at your head and you butt it back with your forehead, you can almost feel the smack of it against your skin (Grossman 2009).
Similarly, Steven Spielberg, brought in as a high-profile celebrity to demo Project Natal at the E3 Expo (June 2-4 2009), comments:
The technology recognized me as a full person…It identified me, my legs, my arms, all of my movements, not just my wrists and my fists, and my thumbs, which is the current state of the art. This recognized my entire person and in a way accepted all of me as a competitor inside the gamespace…It was the most tactile experience I’ve had so far in a gaming space…I got a sense that I was inside the space more than I have on any other platform (Spielberg quoted in Grossman 2009).
As more intuitive and seamless interfaces enable our physical movements to be mirrored by our digital avatars in real-time, the close identification we have with our conventional mirror image may very well extend to include our new virtual alter egos. While all experience, whether physical or virtual, can only be felt by the physical body, the self-avatar assemblage, particularly if augmented with haptic sensory feedback, provides a “mixed reality” experience where sensation, agency and affect are distributed between our offline and online bodies. This “mixed reality” experience is part of an emerging paradigm of the 21st century where our experience of the world around us is increasingly being mediated and augmented by digital technologies.
Biography
Kathy Cleland is a curator, writer and lecturer specialising in new media art and digital culture. She lectures in the Digital Cultures Program at The University of Sydney. Her curatorial projects include the Cyber Cultures exhibition series, Mirror States, and Face to Face: portraiture in a digital age.
Endnotes
[1] For more discussion of the notion of technological prosthesis, see Maquard Smith and Joanna Morra’s introduction to The Prosthetic Impulse (2006) which explores a variety of different conceptions of the prosthetic, from Freud’s proclamation that our technologies will make us “Prosthetic Gods”, to Donna Haraway’s cyborg. Also see Zylinksa, J. (Ed.). (2002) The cyborg experiments: the extensions of the body in the media age.
[2] Ihde describes three sets of distinguishable human-technology relations: embodiment relations, hermeneutic relations and alterity relations: “At one extreme lie those relations that approximate technologies to a quasi-me (embodiment relations). Those technologies that I can so take into my experience that through their semi-transparency they allow the world to be made immediate thus enter into the existential relation which constitutes my self. At the other extreme of the continuum lie alterity relations in which the technology becomes quasi-other, or technology “as” other to which I relate. Between lies the relations with technologies that both mediate and yet also fulfil my perceptual and bodily relation with technologies’ hermeneutic relations” (1990:107).
[3] Brugger explains the shifting phenomenological point of view which distinguishes these three autoscopic phenomenon in the following summary: ‘In an autoscopic hallucination, the subject’s perspective is clearly body-centred, and the hallucinated image evidences left-right reversal …In heautoscopy, a right-handed person’s doppelgänger is right-handed as well (i.e. postural-kinaesthetic information is projected on to the hallucinated image). The observer’s perspective is still mainly body-centred, but a partial projection of bodily feelings into the doppelgänger may lead to an unstable localisation of one’s real self. …In an out-of-body experience, the observer’s perspective is entirely transferred to the reduplicated body which maintains it original sidedness.’ (Brugger 2002: 183).
[4] The experience of flow is associated with the following characteristics (although not all are necessary for flow to be experienced): 1. A challenging activity that requires skills; 2. The merging of action and awareness; 3. Clear goals; 4. Direct feedback; 5. Concentration on the task at hand; 6. The sense of control; 7. The loss of self-consciousness; 8. The transformation of time (Csikszentmihalyi 1991).
[5] More sophisticated movement mapping technologies such as datagloves, datasuits and treadmills are used in fully immersive virtual reality or performance environments. Full-body motion capture suits or armatures can also be used to map the movements of the physical body onto the virtual body in a more natural and intuitive way. In this scenario the individual typically wears a set of markers, one on each joint to identify the position and motion of the body and those movements are then mapped onto the digital avatar representation. In Company in Space’s performance of Cybernetic Organism 3 (CO3), the dancer Hellen Sky wears an exoskeleton which acts as a motion capture device to map her movements onto a series of digital avatars that are projected on a series of screens enabling the performer to perform a mixed reality pas de deus (Bartleme 2001). Performance capture is a further development of this technique, where both body movements and facial expressions are recorded and transferred from live human actors onto virtual characters. This technique is used in some video games and also in films, for example the animation of the Gollum character in The Lord of the Rings films and the animation of the characters in The Polar Express (2003) and Beowulf (2007) and most recently in James Cameron’s Avatar (2009)
[6] http://www.oki.com/jp/FSC/vc/en/bbe/index.html
Bibliography
Bailenson, Jeremy N., and Jim Blascovich. “Avatars.” In Encyclopedia of Human-Computer Interaction, edited by William Sims Bainbridge, 64-68. Great Barrington, MA: Berkshire Publishing, 2004.
Barlow, John Perry. “Being in Nothingness: Virtual Reality and the Pioneers of Cyberspace.” Mondo 2000, 2000.
Bartleme, Edwina. “ Co3 … Stepping into the New Terrain: Company in Space.” Eyeline, no. 47 (2001).
Blanke, Olaf, Theodor Landis, Laurent Spinelli, and Margitta Seeck. “Out-of-Body Experience and Autoscopy of Neurological Origin.” Brain 127 (2004): 243-58.
Blascovich, Jim. “Social Influence within Immersive Virtual Environments.” In The Social Life of Avatars: Presence and Interaction in Shared Virtual Environments, edited by Ralph Schroeder, 127-45. London: Springer-Verlag., 2002.
Brugger, Peter. “Reflective Mirrors: Perspective-Taking in Autoscopic Phenomena.” Cognitive Neuropsychiatry 7, no. 3 (2002): 179-94.
Bukatman, Scott. Terminal Identity: The Virtual Subject in Postmodern Science Fiction. Durham: Duke University Press, 1993.
Castronova, Edward. Synthetic Worlds: The Business and Culture of Online Games. Chicago: University of Chicago Press, 2005.
Clark, Andy. Natural-Born Cyborgs: Minds, Technologies, and the Future of Human Intelligence. New York: Oxford University Press, 2003.
Clynes, Manfred E., and Nathan S. Kline. “Cyborgs and Space.” Astronautics, no. September 1960 (1960).
Cooley, Charles Horton. Human Nature and the Social Order. Revised Edition ed. New York: Scribner’s Sons, 1922.
Csikszentmihalyi, Mihaly. Beyond Boredom and Anxiety. 1st ed. San Francisco: Jossey-Bass Publishers, 1975.
Flow : The Psychology of Optimal Experience. 1st. Harper Perennial ed. New York: Harper Collins, 1991.
Dixon, Steve. Digital Performance: A History of New Media in Theater, Dance, Performance Art, and Installation. Cambridge, Mass.: MIT Press, 2007.
Ehrsson, H. Henrik. “The Experimental Induction of out-of-Body Experiences.” Science 317 (2007): 1048.
Fink, Jeri. Cyberseduction: Reality in the Age of Psychotechnology. New York: Prometheus Books, 1999.
Gallese, Vittorio. “The Roots of Empathy: The Shared Manifold Hypothesis and the Neural Basis of Intersubjectivity.” Psychopathology 36, no. 4 (2003): 171.
Gibson, William. Neuromancer. London: HarperCollins, 1993.
Gordon, Devin. “Polar Expedition.” Newsweek, October 25 2004.
Gray, Chris Hables. Cyborg Citizen: Politics in the Posthuman Age. New York: Routledge, 2001.
Grossman, Lev. “Microsoft Whacks the Wii: A First Look.” Time, 2 June 2009.
Hansen, Mark B. N. Bodies in Code: Interfaces with New Media. New York: Routledge, 2006.
Haraway, Donna Jeanne. Simians, Cyborgs and Women: The Reinvention of Nature. London: Free Association, 1991.
Hayles, N. Katherine. How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics. Chicago, Ill.: University of Chicago Press, 1999.
Ihde, Don. Technics and Praxis. Dordrecht, Holland: D. Reidel Publishing Group, 1979.
Bodies in Technology, Electronic Mediations ; V. 5. Minneapolis: University of Minnesota Press, 2002.
Jabbi, Mbemba , Marte Swart, and Christian Keysers. “Empathy for Positive and Negative Emotions in the Gustatory Cortex.” Neuroimage 34, no. 4 (2006): 1744-53.
Kerckhove, Derek de. “The New Psychotechnologies.” In Communication in History: Technology, Culture, Society, edited by D. Crowley and P. Heyer, 267-72. New York: Longman., 1991a.
Kerckhove, Derrick de. “Communication Arts for a New Spatial Sensibility.” Leonardo 24, no. 2 (1991b): 131-35.
Kozel, Susan. “Spacemaking: Experiences of a Virtual Body.” In Routledge Reader in Dance Studies, edited by A. Carter, 81-88. London and New York: Routledge, 1998.
Lacan, Jacques. Écrits: A Selection. London: Tavistock Publications, 1977.
Lenggenhager, Bigna, Tej Tadi, and Olaf Blanke. “Video Ergo Sum: Manipulating Bodily Self-Consciousness.” Science 317 (2007): 1096-99.
Madrigal, Alexis “How to Use Neuroscience to Become Your Avatar.” Wired, December 2 2008.
McLuhan, Marshall. Understanding Media: The Extensions of Man. 1st Sphere ed. London: Sphere Books, 1967.
McLuhan, Marshall, and Quentin Fiore. The Medium Is the Massage. New York: Bantam, 1967.
Merleau-Ponty, Maurice. Phenomenology of Perception. Translated by Colin Smith. London; New York: Routledge, 1962.
The Primacy of Perception : And Other Essays on Phenomenological Psychology, the Philosophy of Art, History, and Politics, Northwestern University Studies in Phenomenology & Existential Philosophy. [Evanston, Ill.]: Northwestern University Press, 1964.
Mouras, H. , S. Stoléru, V. Moulier, M. Pélégrini-Issac, R. Rouxel, B. Grandjean, D. Glutron, and J. Bittoun. “Activation of Mirror-Neuron System by Erotic Video Clips Predicts Degree of Induced Erection: An Fmri Study.” NeuroImage 42, no. 3 (2008): 1142-50.
Nash, Adam. [Post-Avatar] Properties of Virtual Performance post-avatar archives, Tuesday, 19 August 2003 19 August 2003 [cited 14 June 2005]. Available from http://yamanakanash.net/pipermail/post-avatar.
Ponseti, J, H.A. Bosinski, S Wolff, M Peller, O Jansen, Mehdorn H.M., C Büchel, and Siebner H.R. “A Functional Endophenotype for Sexual Orientation in Humans.” Neuroimage 33, no. 3 (2006): 825-33.
Popper, Frank. Art of the Electronic Age. Paperback ed. London: Thames and Hudson, 1997.
Ramachandran, V.S. Mirror Neurons and Imitation Learning as the Driving Force Behind “The Great Leap Forward” In Human Evolution 2001 [cited 5 November 2007]. Available from http://www.edge.org/3rd_culture/ramachandran/ramachandran_p1.html.
“The Neurology of Self-Awareness.” In Edge, 2007.
Richardson, Ingrid. “Telebodies & Televisions: Corporeality and Agency in Technoculture “, University of Western Sydney, 2003.
Rizzolatti, Giacomo, and Laila Craighero. “The Mirror-Neuron System.” Annual Review of Neuroscience 27 (2004): 169-92.
Schroeder, Ralph. The Social Life of Avatars: Presence and Interaction in Shared Virtual Environments, Computer Supported Cooperative Work. London: Springer-Verlag, 2002.
Sermon, Paul. “Puppeteers, Performers or Avatars – a Perceptual Difference in Telematic Space.” Paper presented at the Perceptual Difference, Perth, 8 September 2004.
Sharir, Yacov. Virtually Dancing n.d. [cited 8 November 2007]. Available from http://www.art.net/~dtz/sharir1.html.
Slater, Mel, Daniel Perez-Marcos, H. Henrik Ehrsson, and Maria V. Sanchez-Vives. “Towards a Digital Body: The Virtual Arm Illusion.” Frontiers in Human Neuroscience 2, no. 6 (2008).
Smith, Marquard, and Joanne Morra, eds. The Prosthetic Impulse: From a Posthuman Present to a Biocultural Future. Cambridge, Mass.: MIT Press, 2006.
Sobchack, Vivian Carol. Carnal Thoughts: Embodiment and Moving Image Culture. Berkeley: University of California Press, 2004.
Stephenson, Neal. Snow Crash. New York: Bantam Books, 1993.
Stone, Allucquère Rosanne. “Split Subjects, Not Atoms; or, How I Fell in Love with My Prosthesis.” Configurations 2, no. 1 (1994): 173-90.
Taylor, T. L. “Living Digitally: Embodiment in Virtual Worlds.” In The Social Life of Avatars: Presence and Interaction in Shared Virtual Environments, edited by Ralph Schroeder, 40-62. London: Springer-Verlag, 2002.
Thompson, Clive. “Victory in Vomit: The Sickening Secret of Mirror’s Edge.” Wired, 16 November 2008.
Zylinska, Joanna, ed. The Cyborg Experiments : The Extensions of the Body in the Media Age, Technologies (London, England). London: Continuum, 2002.
Associated Online Sites
YouTube video: Olaf Blanke describing the rubber-hand illusion – http://www.youtube.com/watch?v=TCQbygjG0RU
YouTube video — Olaf Blanke describing a virtual Out-of-Body experience: http://www.youtube.com/watch?v=4PQAc_Z2OfQ
Paul Sermon, Telematic Dreaming: http://creativetechnology.salford.ac.uk/paulsermon/dream/
Stahl Stenslie, CyberSM and Interskin: http://www.stenslie.net/stahl/
OKI Electric Industry’s FaceCommunicator: http://www.oki.com/en/fse/case/bbe/index.html
Microsoft’s controller free Project Natal system for Xbox 360: http://www.xbox.com/en-US/live/projectnatal/
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Australia.