Unity VR - Memory Bubbles

Idea

Memory bubbles is an idea for a VR app that I came up with at a VR hackathon 2019 in Lausane: A Fotoalbum consisting of 360° videos.

 

Usual media displays are in 2D and thus are perfect for classic photos and videos. 360° footage can be displayed by interactive viewers, however the experience is limited: due to the display you always get the 2D vibe.

 

You find the contrary when using VR devices. They are made for 360° content. Once inside VR, you naturally look around moving your head absorbing all the details around you.

 

I imagine this immersive action can be particularly beautiful when looking at past memories: Exchanging the static 2D photo reminder by a fully immersed 360° experience to revive the moments.

Character looking at a memory bubble in an open world photo album.

To realize the idea I am going to use an Oculus Quest2 as VR device and Unity, as it offers a somewhat matured interface for app development on the Oculus.


Character inside the bubble.

Enjoying the 3D view of the real world office.


Getting Started

Demo Projects provided by Unity & Oculus to get rolling with app development.

Unity offers great free material to quickly get into creating something and get a first feeling of the highlevel concepts of app development on the platform.

Together with the not quite so straightforward Oculus integration I eventually got a first unity demo app running on the VR headset within a weekend. Also this video helps get running.

 

Seems 2020 is still a year of active changes for both vendors Unity and Oculus. So during development I ran into multiple compatibility issues and major feature changes*.

In this period neither Unity nor Oculus offers a documentation that would get you from start to end, building some basic knowledge of both sides is required. In December 2020 this video gives a working solution using the latest stable versions/firmware available.

 

*Major feature changes:

Unity 2020 splits the "Asset Store" - its central 3rd party integration point -from the IDE and thus forces users to use the package manager workflow instead.

Also the general XR integration workflow changes with Unity 2020, vendors like Oculus only document the now deprecated that has been used up to Unity 2019.


Memory Bubbles

Let's turn the bubbles idea into technical specifications that can be realized:

 

360°-Video-On-Collision-Sphere:

Memory bubbles shall be spheres that display videos, that can be watched when walking into them, i.e.

  • 360° videos can be watched from the inside of the bubble
  • 360° may only start playing when bubble is focused (e.g. player inside bubble)
  • Content of the bubble may be seen from a distance ("Video Thumbnail")

In 2020 rendering 360° video footage on the inside of a sphere is no standard workflow in unity, it requires 2 issues to be taken care of:

 

1 - In Unity standard UV mapping to a sphere requires content (images, videos) to be provided in the equirectangular or cubemap format. This may require preprocessing of the 360° video footage or scripting a custom UV mapping in Unity.

To collect 360° footage I used the Ricoh Theta V that saves videos in its custom format.  For unity integration I chose preprocessing, i.e. converting the videos into the equirectangular format. Maybe one day I add a custom UV renderer.

360° videos in different formats: Ricoh Theta V camera uses the Double Fisheye format, Unity imports Equirectangular 360 by default

2 -Using Unity's standard renderer videos are only displayed on the outside of Unity sphere primitives (it uses backculling, and the spheres normals point outwards). Thus it is required to either create custom spheres with normals pointing inwards or script a custom renderer.*

I created a custom, high resolution UV-sphere in blender with inverted normals and ported it to Unity using Unities preferred .fbx format.  The spheres are simple static objects in the unity scene, so there is no export magic to consider.

low resolution UV spheres in blender: left with normals pointing outwards, right with flipped normals pointing inwards

Now with footage and sphere in the right format we can setup the  360°-Video-On-Collision-Sphere with a point light at the spheres center and Unity standard components added to the sphere's mesh object:

  • Mesh Renderer & Mesh Filter
  • Video Player
  • (Capsule) Collider
  • Script (to connect the above three)

The video (or its URL) can be set with desired details in the inspector, as Render Mode we use Material Override and select the sphere's Mesh Renderer as subject. There are less straight forward ways using a Render Texture, but we don't need it. Than select Is Trigger in the colliders menu since the collider is not meant as physical property, but only to trigger the video. Finally add a script defining the VideoOnCollision class:

 

public class VideoOnCollision : Monobehaviour

{

void Start() {}

void Update() {}

public UnityEngine.Video.VideoPlayer video;

 

void OnTriggerEnter(Collider other)

{

    video.Play();

}

 

void OnTriggerExit(Collider other)

{

    video.Pause();

}

}

 

and connect the Video Player.. Voilà the memory bubble is ready.

 

First memory bubble prototype integrated in scene. To move in the scene I used a FPS character as explained by Brackeys.

* As an integration of 360 video player in VR content is very common, it may be surprising that this is not considered a standard workflow. However Unity's skybox offers the a quick&easy 360° content display, as explained on Unity's official channel. In my application I want to smoothly walk in and out the sphere, so using the skybox is not a solution for the bubbles.


My first Oculus App

Share first prototype for oculus

Maybe next:

- use url & stream to host videos outside app