Strings Attached
By Nikita Kayal & Roger ter Heide
Strings Attached is about control and identity in VR. What happens to your feeling of embodiment and immersion when someone else takes control of your movements or your appearance.
We have been playing with embodiment within VR for a number of years. One of the things, that is interesting is how you identify yourself with your own avatar, and how that creates immersion. In Strings Attached we asked ourselves if it is possible to partially take control of a person’s avatar without breaking this feeling of embodiment and immersion. We would like to explore if we can reach a feeling of harmony and flow for the controlled player, or if it will lead to struggle and conflict. For that, we created a number of experiences to test our ideas.
We split our research in strings attached into 3 different approaches. The first focuses on sharing bodily controls or puppeteering, where you can take control of the limbs of the person’s avatar. The second is on projecting your voice on the avatar and looking at how that influences the person in VR. The third is playing with a dissociative mirror, where the image in the mirror can be influenced by the player outside of VR.
Our research on movement and interaction
After working on multiple projects around dance and performance, we wanted to further explore our insights on movement, embodiment, and control using the perspective of gameplay. We wanted to expand our understanding and ideas on puppeteering or other forms of interaction that were not always interesting from the perspective of dance.
Sharing control
Initial ideas and controls
Our core concept was to explore new forms of blended control. One player is in virtual reality embodying a virtual body, the other player outside of VR exerting control over this same virtual body.
Our first challenge was to define, how can we best achieve such a control? How does it feel to be embodying a body that only has partial control? At what point does the embodiment break and make you look at your body from a third-person view? What happens when you can respond real-time to the other player’s influence in warping your movements? When does the influence start to become something that feels like a bug? We set out to look for answers to these questions with a series of experimental prototypes.
Initial trials and extreme forms of controls
Our initial trials had something that felt close to a string puppet, but the control was so extreme that it quickly broke immersion. The question we needed to ask ourselves was what elements of a string puppet work for our experience and what elements don’t? A string puppet typically continues to exert control even when there is no active input from the player — as the player moves themselves, they automatically might pull on the strings. By pulling the strings in a certain direction the puppeteer could pin the VR player’s avatar to a certain position, often breaking the immersion. It felt more like a medieval torture device than a journey into shared control.
A second insight was that the way we mapped the controls to keys on the keyboard, the controls were getting too complex. In our initial research, we had looked at the game QWOP where you control a ragdoll runner using a keyboard and try to make him go as far as possible. When we relooked at it through the lens of control, we realized that because we were playing multiplayer, such a complex interaction wouldn’t work. The non-VR player would keep struggling with the controls. This prevents a shared dialogue about partial control of the body.
The mapping of the controls needed to allow the puppeteer to have a natural feel of control. The controls should create more subtle forms of movement and change in movement so the VR player could experience the virtual tug on their hands without feeling disembodied. What we also needed to do was to limit the amount of movement the puppeteer could create. Since there was a natural limitation on the length of the arms of the VR player, we also wanted to limit the influence of the puppeteer, creating a balance in dynamics. These were interesting perspectives and parameters we could gauge our controls with when we adapted them.
Staying Embodied
There were no best practices or established techniques on how to improve the experience, so it took us some time to figure out how to achieve the right interaction. First, we mapped the controls to the gamepad to create an analog form of control. As a second step, we changed the perspective of control by making the influence relative to the player’s current position instead of fixed in space. Third, we limited the extent of influence the external player can have to a certain range. These key changes made the blended control more believable.
We now had an experiment that worked for our goals. You could stay immersed and embodied as a VR player and still experience the tug or movement by the external player.
The same was true for the external player. The external player could now get a feel of orientation and control with respect to where the virtual hands currently were.
One good sign that we were succeeding was that the need to communicate verbally diminished and we could respond to movements instead of needing to verbalize our intentions.
Balancing Control
We added the possibility to balance the amount of control exerted by the puppeteer versus the person in VR. This allowed us to find out how the experience changed as it went from full control by one player to full control by the other. We found that there was a Goldilocks zone of shared control. If we set the balance between 30% to 70% both players could still feel engaged. If one of the players had less than 30 percent control it led to a feeling of irrelevance.
Emergent Behavior
We saw interesting interactions appear due to the asymmetric nature of our setup:
· The external player only had control of the arms, they could choose when they wanted to intervene or not, and how dominant they could be.
· The VR player could move their entire body. So, they could turn or step away, to change the impact of the external player’s control.
Observations
· The VR player could observe the tug and follow the intent of the puppeteer. Since the puppeteer no longer easily observed a direct effect, they suddenly felt a loss of control.
· A VR player tried to pull their virtual hands in an attempt to counteract the tug or the pull from the puppeteer. While this feature wasn’t supported, the puppeteer could take that as a cue and choose to respond in different ways.
· A VR player was flapping their hands to indicate their intention and direct the other player. The interesting thing was that communicating the intent did not mean that the other player would comply.
· Some external players felt the urge to convey their control through larger movements of their own body. Sometimes mimicking what they wanted with their body.
· It needs some time for both players to understand what it means to blend their control. They need to first understand each other’s movements before they can explore more nuanced expressions.
Player feedback
· Some other non-VR players wanted to explore the possibility of switching from a game controller to larger movements to act out their puppeteering role. Currently, their movement is limited to the subtleties of their thumb.
· One of the players remarked that they didn’t wish to exert control as a puppeteer when they saw that the VR players were moving their body themselves. They waited till they settled and were ready for an interaction with shared control.
· Another player wanted an RPG level of control over the avatar and wanted the “puppet avatar” to respond more to their input.
I think this is an interesting start to a discussion on how people identify and acknowledge the human behind virtual bodies.
We also concluded that if players haven’t experienced a lot of avatar embodiment it is more difficult to experience the change in the embodiment. So players new to VR should be allowed a longer time to experience their embodiment without it being influenced.
Adding a context
We added a simple painting interaction with some paint buckets and a canvas. The painting would help us explore the possibility of flow or conflict. Initial observations with this prototype seemed promising. Using a blank canvas seemed very open ended. The two players would either need to agree on a common goal, or one would need to submit to the other. But most of the time the different intentions of each player never became clear while painting.
We then added a partial silhouette of an avatar face to the canvas. We also added a mirror, suggesting the players attempt to make a self-portrait. The players could then still have their own ideas on how they would approach this. It gave sufficient context to make sure that the players could start to sense and understand the intention of the other player.
This enabled us to experiment with the idea of harmony and flow versus conflict. We found that it was much easier to be in a state of conflict than to find the magic moments of flow and harmony. Once players had a goal and an idea in their mind, it was more difficult to let go or follow the direction of the other player’s movement. Frustration was more common in a collaborative canvas space. The amount of control was in more ways asymmetric in nature, since the VR player had the option to step away from the canvas to stop the possibility of painting. It was interesting that there was a level of agency that seemed natural for the VR player to explore.
The identification of the avatar was almost always associated with the VR player as both players normally acknowledged more agency being present for the VR player.
Ragdoll
As one of the small experiments, we tried embodying a ragdoll. The hands and the heads were attached to the VR controls and the rest of the body was calculated using a physics ragdoll system. It was interesting to have the feeling of a floating body, but it was difficult to stay embodied. It was also extremely difficult to control even without any influence from the puppeteer. So in the end the ragdoll physics would interfere with any subtleties of control between the VR player and the non-VR player.
Varying control using music
Initially, we varied the amount of control with a button. The puppeteer could change the ratio of their influence on the avatar. We asked ourselves what happens if it varies with music? How would the players react to changes in the music?
We used the amplitude of the music. The results were interesting. Firstly, moments of free movement for the VR player were interspersed with heavy tugs in between. This reduced the possibility for the VR player to get used to the body and find a way to have a dialogue with the puppeteer. The constant change in the ratio of control resulted in a loss of understanding of what was going on. There wasn’t sufficient time to understand. It led to frustration and mostly conflict. On the other hand, you could anticipate the amount of influence you had because of the music, and might just let go and let the other player perform the movements while you waited for your turn.
We tried different kinds of music but that did not make much of a difference. There were too many factors influencing the amount of control. In some ways it was similar to the ragdoll, the added layer took away the possibility to understand and explore the subtleties.
As it didn’t afford the experiences and dialogue we were intending and we didn’t explore further.
A context with discrete choices
In another prototype, we changed the environment to provide the players with discrete choices. We wanted the players to quickly learn what those choices meant. So we added floating fishlike objects, that after touching would change the tempo of the music being played. The color of the fish indicated if the tempo would go up, down, or reset. So players could decide how they want to control the tempo of the music being played. The non-VR player could try to influence what color object the VR player would touch.
Since you could touch the fishes with both hands, the fishes were at different heights, and the movement space was much bigger than the more wrist-level movements with the canvas.
We saw that while the audio choices were discrete, the motivation to make a certain decision wasn’t that high. Unlike the canvas in the painting space, where players had specific ideas and a strong motivation to follow through, the choices here didn’t have a consequence they felt invested in. Even if the other player would try to enforce a different choice, it was easier to let go. It did not feel as intrusive as someone drawing over or spoiling your painting. One more interesting thing was that if you focused on controlling one hand, the other player could easily counteract the effect with the other hand.
Playing with the tempo of the audio was a great step to transition into the next experiment.
Projecting voice
What if you could influence the other player in VR with your voice? There were multiple questions we explored within this. Do players sing? Do we just ask them to say something? Or do we just want people to make sounds? Since the other player would hear and embody it, we wanted it to be more than just sounds. We also thought that not everyone might be comfortable singing or coming up with things to say on the fly. After some brainstorming, we decided on using poetry and spoken word pieces that non-VR players could narrate.
The next step was to explore what the embodiment and the environment would look like. We had a simple idea of a voice rippling through the body and the space around the body. Our basic idea was to translate the audio amplitude to a ripple on the avatar's body. We had some challenges in getting this right and making it modular. We also wanted to be able to extend the ripple to other objects in the environment. There was one technical limitation. The voice audio of the non-VR player could only be processed with a delay of 1 second.
When we tested this experience, it was interesting to see how tempo and volume could project on the avatar especially for the non-VR player. However, for the VR player
We felt the core experience missed some interaction elements.
The voice was an indirect form of influence, it influenced the shape and form of the VR avatar but not the interaction with the space around.
To start with, we added objects the players could interact with around them. When the form of their avatar is enlarged because of the spoken words, it could change the body size and lead to unintended interaction. However, it seemed like a very crude interaction. It also made the spoken word or poetry lose its meaning and context in the experience.
To increase the role of the audio in the interaction, we added possibilities to create echoes of what the player said. The VR player could touch interactive objects that would also ripple based on the audio and create an echo of certain words. So the VR player could now choose moments and words they resonated with by interacting with the objects.
While the interaction was interesting, the projection of the words didn’t translate into the actual movement of the avatar and focused on a balance of control. The voice interaction got externalized as the players focused on making echo-like patterns. It seemed like the interaction was defined by how players projected their own bodies instead of an inward focus on how it influenced their own embodiment.
Another observation was the role of the speaker. In our current setup, the world would stop responding and creating ripples if the speaker was silent or paused. For the VR player there was a desire for constant input, to enable and echo, as silence was almost like a punishment.
We would like to further explore the way emotions, audio, and movement can interact, as it feels as if we just scratched the surface.
The dissociative mirror
One of the last experiments we did was based on a dissociative mirror. In a simple environment, we focused on the player’s avatar, and a mirror. We played with the concept where the mirror image follows the avatar, most of the time, but can also differ. The non-VR player could “play with time” and change the way the mirrored avatar behaved. At first, we let the non-VR player expose the VR player to different effects on the mirror image like slow-motion, moving with a delay, or going backward in time.
We also tried the reverse. So here the avatar that previously was exactly following you would have a delay or move in slow motion whereas the mirror image would follow your real body movements. So, it would suggest that your avatar was actually following the mirror image. This was a very unusual quirky effect to experience as a VR player.
The non-VR player's actions were limited to choosing the effect the mirror would display or leaving the mirror in a normal state.
We had several questions. Would the VR player still identify this mirror image that mostly follows but not always? Would they see it as a mirror image, or would it break immersion completely? We saw that because this meme has existed in pop culture, it wasn’t as difficult to maintain immersion, our brains could label it as a magic/fake mirror. We saw that a movement going backward in time was immersion-breaking. It was difficult to understand as it felt like the mirror was just doing something else. Maybe if the movement was timed at a specific recognizable moment, it might be more understandable. Slow-motion or a delay was more relatable and coherent for both players.
Initially, we struggled to give the non-VR player meaningful participation in the experiment. Initially, while we were still exploring the possibilities we had a binary key control, meaning as the non-VR player pressed a button a chosen effect was activated for a fixed time, in this case, 5 seconds. A big limitation here was that the non-VR player needed to wait till an effect finished before they could choose the next effect. So, the non-VR player could not respond directly to the VR Player’s actions. Another disadvantage was that the VR players often wanted a much longer time period to explore a certain effect or play with that effect.
As a refinement, we changed the way the control of time worked. We decided to focus on the effect of delay and allow the non-VR player to continuously adjust this using the analog stick of a game controller. This way the player could increase or decrease the delay of the mirror image instantaneously. But they could also flip the delay to the avatar instead of the mirror image. Although the amount of direct control on movement was limited, for the non-VR player it was interesting to see how you could play with the expectations of the VR player, sometimes indulging in their explorations, and sometimes reversing the effect.
An interesting question that came up during our experiments is how you identify with the mirror image. It sometimes felt that the mirror image was a copy of yourself and sometimes someone else. When the delay was small it felt like another person that was mimicking your movements. When the delay was large it could feel like the other character was slow in understanding you, or it could feel more like you were playing with a recording of yourself, allowing you to create patterns and interactions with your past self.
Throughout the experiments, we have seen emergent play, in this test with the dissociative mirror, HKU students are projecting the mirror behavior to the real world
Reflections on exploring control and flow
Our experiments have explored how your embodiment and immersion change as you play with control. It is easy to create conflict and break immersion when you interfere with someone’s control of their avatar. But we have seen that there is an interesting realm of mixed control to explore.
Direct control of the embodiment
For direct bodily control, we see that creating flow or conflict depends much on the players. Conflict is easy to achieve, by having separate goals and overpowering the other players’ intentions. Within conflict, it is interesting that players can find ways to rebel. When the non-VR player dominates the avatar’s arm movements, the VR player can still walk away or turn away in VR and change the effect it has. Even more interesting is that the VR player can also rebel by playing along. This was done by following the nudges of the non-VR player and thereby diminishing the feeling of influence and agency for the non-VR player.
For flow, there are different ways to explore it. It can be an explorative attitude, where players observe and the non-VR player subtly introduces interactions the moment the VR player is in a non-active mode. It can be a playful mischievous mode where players playfully flip the other's intentions without ignoring them. What is especially interesting is when the VR player not only acknowledges the influence but goes with it, as it can create a subtle dance of who is in the lead.
Key research question: As a VR player can you still have a feeling of harmony and empowerment and presence, as the external control on your avatar increases?
Answer: Yes, within certain bounds. It is important that you leave some agency to the player, but it is important that the players show respect for each other’s choices.
Projecting voice
It was interesting to explore how audio, and specifically a voice can influence you as a player. The asymmetry in control is even larger here. Where the non-VR player can only project through their voice and the way they choose to play with the poem we provided. And the VR player chooses how to move and what audio to reflect on. As the poem reader, you can let yourself be influenced by the movements and offer the VR player different patterns that translate to visualization by changing your tempo or choosing what words to accentuate. Adding the possibility to echo some of the words was essential to increase the agency of the VR player. In this case, one of the limiting elements was that without the voice of the non-VR player the agency of the VR player was limited.
One of the things we missed was that the person’s voice did not directly control the way you moved. It affected the visualization of the body but not the control of the body. It did allow for some dialogue, but not in terms of movement. So exploring how the voice could directly affect the VR players ability to move would still be something we would like to further explore.
The dissociative mirror
It was interesting to see how a meme from popular culture can lead to inspiration and play. The simplicity of the concept was appealing and easy for a larger audience. Where the idea of joint control of limbs can be alienating, the dissociative mirror is much more easily accepted, although there is an important shift. As soon as we see the delayed mirror image we start associating it with another person, who is mimicking you. It can lead to play where you almost challenge your mirror image to follow suit, to a feeling of connection when the movements are slow and subtle and the delay is not too long. For the non-VR player the interaction is very subtle: choosing to offer reflection of the person’s avatar or playing with the concepts of mime or the children’s game “Simon Says”.
Key Research Question: Can you still identify with your avatar and its mirror image when movements and identity change outside of your control?
Answer: Yes, yes but the projection can feel weird, and the identification can flip when you play with time and delay.
Conclusion
We are very thankful ImproVive got the chance to explore this domain with the help of the grant by the Creative Industry Fund NL. It has inspired us and others to show there is an interesting world out there to further explore in VR, where we can play with the boundaries of embodiment, but where we can also learn what it means to be truly open to the input of others to create a feeling of joint flow and purpose.
