Maximising VR immersion through acting

Imagine that you are in a VR adventure where you enter a room full of characters. They are all minding their own businesses: talking to each other, playing games and, in general, being alive. When you approach a group, or a single character, you start talking with them normally and they talk to you like if you were one of them. They are all following more or less a script for a story where you are the protagonist, and it is through interacting with all characters how you can progress in your very personal adventure. How could we create this?

The approaches

The applications for social interaction in VR are well known and one can find plenty of great VRSocial experiences out there where the users, represented by an avatar, can connect with other humans and interact. They are very compelling experiences and certainly I have had some magic moments playing Rec-Room or Star Trek Bridge Crew… but in order to have the “perfect”adventure  experience everyone needs to stick to their role, never break the character and be part of a story where they guide you (the protagonist!) through your quest… Also might need dozens of characters, all of them humans agreeing to play for you. Certainly there are some problems using this approach.

One could also create a system where all the players are controlled by an AI using advanced voice recognition and dialogue creation. I see many people being attracted by this idea, but the technical challenges are huge and I don’t expect them to be solved in the next years. Sure you can ask your phone “OK Google, what is the weather like?” but that does not mean that you can have entire, deep, believable conversations with it… and more times than not these type of interactions end up in frustration from the user.

That frustration, if it where in VR, instantly leads to a decoupling from the experience and a lost of the immersion. Although one could mask it with some clever design like: you are in a foreign country and the locals can barely speak your language or you can just talk to animals and they make gestures. The limitations of this approach are also obvious and difficult to overcome.

My idea here was to mix both soltutions: what about making all characters be controlled by an AI when they are not engaging with the user, but have one single actor play all of them when interaction is needed? The actor could have a dossier for each character: with some background to explain the motivations, information knowledge, etc of the character and some tools to animate the “possessed” body, alter the pitch of the voice or express emotions.

The player would feel that each character is intelligent and unique, they will feel the real protagonist of the story and, ideally, be totally immersed into the experience.  Interestingly enough the very active role of the actor could be turned into an enjoyable challenge as well: “can you improvise over a dozen characters?” and maybe make it a game that work both ways where you chose if you want to be the main player or literally everyone else, each track would have its own rewards.

The Magic Forest

I really wanted to give this idea a try, and the perfect moment was during the Future Artists VR Lab:

Sadly I could not attend the whole 4 days, which left me with just ~10 hours to assemble a team, propose my idea (and hope it was accepted) and develop it! Upon arrival I met the incredible Kayleigh (@kayleighsmyth96),  Sam (@videosmithery) and Jamie (@QueerMediaUK) who agreed to give it a spin and proceeded to discuss ideas on how to make something that could do the job within the time frame using our combined talent the best possible way.

The Magic Forest is a short Vive experience that happens within a pond in a deep forest. In the centre of the pond there is a magic mirror that reveals the true soul of the player, but to reach there the user needs to please the three mighty Aquardians , protectors of the lake!

Each Aquardian will talk to the player and try to explore who they truly are. Progressively engaging into deeper conversations that explore the fears, passions and strengths of the players respectively. Once the blessing of an Aquardian is granted, a rock will appear in the water that allows the player to advance a little closer towards the mirror… and once the mirror is reached the player will see their true face based on the responses they gave.

The nuts and bolts

The technical side of the project was developed using Unity (myself) and Tilt-Brush (Kayleigh), while Sam took care of shaping the story and Jamie did the acting.

I have to say I was impressed on how fast and easy was to import the Tilt-Brush data. Although it can lead to some performance problems (the experience was running at 50fps) the amount of assets that can be created in such little time is amazing and a great fit for these type of events!

But let’s dive a bit more in the Unity side. The experience is actually compelled by two applications: the VR experience itself, running on a PC with a Vive; and the actor tool which was exported as an Android app that the actor can run on their phone.

The core of the interaction was done through conversations. I used Photon Voice for this purpose, allowing me to quickly hook phone and PC together in a chat room, then with some AudioMixer magic in Unity I could create different effects for each Aquardian so the actor wont have to worry about doing any voice acting at all.

magicForestApp

The app supports not just chat with voices, but also needs to provide the actor with buttons to cast their voice through each character, a live mini-map to know where the player is looking/talking to, more buttons to enable/disable the rocks and finally to change the reflection the player will get in the mirror. I usually would have done this with socket programming but having Photon already there, and given the limited time-frame, I just RPC it all the way.

All of this is public in my GitHub !

 

Conclusion

At the beginning the team was not sure about the value that the app would add to this, but soon it was obvious that this was a powerful tool and Jamie got the hang of it very quickly. The actor was isolated in a different room to maximise the effect, and Jaime did an stellar job improvising and changing personalities, having conversations (even between the Aquardians!) and bringing a great mix between drama and comedy while pushing the player feelings.

When I first tried what we had created I had a hard time believing that this was all managed by a single person, the acting plus the voice modulation was amazing and soon all players got sucked into the experience.  The demos were done at the VRLab event, with around 40 people watching… but something magical happened there as even the shier players got more and more immersed in the conversations and ended up revealing quite private thoughts in front of everyone. Some of the players engaged in very deep conversations while others had ridiculously hilarious ones and ended up in tears (really!). Many were amazed when they saw their reflection and others were a bit mad at Jamie when he picked the “evil” reflection for them, but in general I think everyone had a very personal experience and told us how immersive it all felt so I am happy to say that the experiment was a massive success and this is definitely a route I want to explore further!