Oceanscope is an immersive VR game experience.
It serves as an experiment to marry science education and immersive experience. The final project serves not as an end, but the start of a series of further quest: how can better engage curiosity by creating a virtual space that encourages exploration? What are the key concepts to pay attention to children engaging in virtual reality? How can we think out of the box to create new interactions?
Technical and Educational Research, Modeling, UX/UI, Sound Production, Game Development
Genre: VR Game
Unity, Maya, Google Blocks, Adobe Audition, Figma
Duration: 3 months
Currently, I am carrying out research, design, and development all by myself. I try to tackle these questions holistically.
Base on my research on the application of new media in children’s education, I had in my mind a tangible user interface(TUI) platform that aims to encourage exploration and understanding of nature and science.
Unlike traditional Graphical User interfaces(GUI) that only allows limited on-screen interaction, I propose to extend the use of a screen with a trackable periscope and projection mapping. Through active engagement in this immersive digital playground, the children can get an idea of species evolution through the case of whales, as well as the climate and geographical factors that are critical to the process. The immersive, game experience encourages the player to empathize with the marine creatures, to look at the world they are living in not simply as a static existence, but a product of a dynamic change in an extended time span.
Nijholt, Anton. Playful User Interfaces: Interfaces That Invite Social and Physical Interaction. Springer, 2014.
This book is a collection of academic papers on the topic of playful interfaces, which focuses on user interfaces to applications that have been designed for social and physical interaction. In other words, this book explores nontraditional human-computer interactions that create recreational and educational experiences catered to children as the target audience. It suggests that by creating a digital augmented real world, interface technology can be evaluated not only by its contribution to efficiency but also by how much it can enhance positive social and physical skills of the players.
Different chapters of the book discuss the application and research background under various topics. Amongst them, interactive playgrounds, child-computer interaction, tangible game interfaces, full-body interaction have informed me the most on creating an installation for educational purposes. Specifically, it has pushed me to evaluate my project from the perspective of gamification, which serves to provide the children an active participatory experience. Besides, the book has introduced methodologies that are crucial to an engaging project experience, namely context-awareness, personalization, and adaptiveness. This book also includes various successful examples of playful interfaces for children, e.g., slides with projections, where children were encouraged to interact with the space intuitively. Thus, the physical interaction in the space has also become the focus of my project, where the player need to physically use force to trigger interaction.
Zimmer, Carl, and Douglas John Emlen. Evolution: Making Sense of Life. Roberts and Company Publishers, 2016.
This book introduces the fundamental concepts of evolutionary theory, such as natural selection, genetic drift, phylogeny, and coevolution. Starting with interesting stories and cases each chapter, this book explains the key concepts with clarity and graphical explanations. As for marine life evolution, the book explained the marine life not only from merely how one species transform throughout time, but also supported the research with the knowledge in paleontology and geology. The book also highlights the connections between species on Earth through the angle of Phylogeny. Today many people are aware of the climate factor that influences the progress of evolution, however, this book also touches upon the idea of “coevolution”, where ecological partners help each other evolve and survive.
This book opened up my mind to the idea of “ocean evolution”. I realized that there are many factors critical to evolution. Besides climate change, the bio-diversity and geographical barriers also play a huge part. I realized that my initial idea of mapping out many marine species was too ambitious. Instead of doing marine life in general, I decided to focus only on whales, a typical marine mammal that initially inhabited in the ocean, developed into a terrestrial mammal and later went back to the ocean.
Bérigny, Caitilin De, et al. “Tangible User Interface Design for Climate Change Education in Interactive Installation Art.” Leonardo, vol. 47, no. 5, Oct. 2014, pp. 451–456., doi:10.1162/leon_a_00710.
The authors discussed how tangible user interface objects can be important educational and entertainment tools in environmental education. The authors describe their interactive installation artwork Reefs on the Edge, which incorporates tangible user interface objects and combines environmental science and multiple art forms to explore coral reef ecosystems that are threatened by the effects of climate change. The authors/artists argue that the use of a tangible user interface in an installation-art setting can help engage and inform the public about crucial environmental issues.
Having read this journal, I am more convinced of the significance of using a tangible user interface rather than a graphical user interface for education. The journal emphasized the role of innovative forms of interactive technology in global education, which confirms my project idea to combine motion tracking and projection mapping to create an immersive space, in order to make the exploration of evolution more engaging.
Kac, Eduardo. Telepresence & Bio Art: Networking Humans, Rabbits, & Robots. University of Michigan Press, 2005.
The book is a personal collection of works and analysis of electronic and bio art by the author, Eduardo Kac. The book was divided into three sections, “Telecommunications, Dialogism, and Internet Art”, “Telepresence Art and Robotics” and “Bio Art”. Overall, they cover works and their respective background in telecommunication media, interactive systems, the Internet, telepresence, as well as the overlap between electronic art and biotechnology.
Focusing on the first section, I am very inspired by several pioneering examples of interactive systems that challenged the traditional view of our human body and the established communication system. Through the variety of projects that use the body as input, the pioneering use of VR headset for performance as well as nuance ways of manipulating images on a normal screen, I get to know the technical experiments that have been conducted for creating an immersive experience.
A-Vovle by Christa Sommerer & Laurent Mignonneau
In the interactive real-time environment "A-Volve" visitors interact with virtual creatures in the space of a water-filled glass pool. These virtual creatures are products of evolutionary rules and influenced by human creation and decision. The human not only decided and created the creatures in the water but can also interact through the tangible interface. The creatures exhibit life traits, such as hunger, death, competition, and reproduction.
Even this project was finished back in 1994, its interactive presentation, as well as its project thesis, are still appealing to me. Throughout the interaction, the human decisions in the creation of a new form and the rules of evolution and selection will create an environment that is open to all possible modifications and selections, following the laws of evolution and creation. It challenged the idea of natural evolution and substituted the natural cycle with an artificial one. This project is a good juxtapose to my project idea, where even both of them focus on Tangible User Interfaces and the theme of evolution, they differ in their effect. A-Vovle targeted on adults, where the creepy experience were effective to communicate the intention behind this project. However, since my project targets on children receiving science education, I want to make the experience more positive and lighthearted.
Genesis by Eduardo Kac
"Genesis" (1998/99) is a transgenic artwork that explores the intricate relationship between biology, belief systems, information technology, dialogical interaction, ethics, and the Internet. The key element of the work is a synthetic gene that the artist invented and that does not exist in nature. This gene was created by translating a sentence from the biblical book of Genesis into Morse Code and converting the Morse Code into DNA base pairs according to a conversion principle specially developed for this work.
During the interaction, anyone with internet access can manipulate the laboratory manipulation at a distance, which explores the cultural and technological influence that people have on evolution today. The potential cause for biological mutation of bacteria, ultraviolet light in the gallery, is in the project controlled by anyone with computer access to the system. This interesting action has intrigued my thought on my project. I also wanted to incorporate something more participatory, fun yet easy mechanism in my project experience.
Plankton Populations Exhibit at Exploratorium
Plankton, the tiny organisms that fill our oceans, produce half of the oxygen we breathe and soak up more carbon dioxide than all the world's rain forests combined. Plankton Populations exhibit models worldwide plankton populations using special "lenses" on a digital ocean display, allowing visitors to come to their own conclusions using real data.
In fact, back in 1997, there was an interactive prototype, metaDESK, invented by Brygg Ullmer and Professor Hiroshi Ishii in the MIT Tangible Media Group, that applied the same interaction as Plankton Populations. Even though the two projects were technically identical, their project experience was completely different. metaDesk was an innovative representation of a possibility for future interface, making the audience feel distanced, while Plankton Populations was inviting to people from a wide age range. In the documentation video, I see both children and adults are fascinated to explore how the plankton looks like in different regions. The interaction of using a “magnifier” to look at planktons’ form is such a clever idea, since it justified the action, and gave the audience an incentive to continue exploring, even some of them just do it because the process is fun. This inventive use of technology probed me to look into existing technologies that can be used differently by my project. Traditionally the VR trackers were used alongside the VR headset, but I want it to live outside of the VR world, and use it as a controller to better assist the audience to interact with a space.
Connecting! Block Town by TeamLab
In Connecting! Block Town, cars and trains run along digitally projected roads and railroad tracks that are connected by physical wooden blocks. As children place different blocks on the table, more cars and trains appear, forming a townscape. Children play by placing the blocks in different positions to create a town collaboratively with other children.
When I first saw this project, I was immediately fascinated by the immediate feedback it gave to the audience responding to their action. It was easy to connect between all the different blocks, but it is the animation that took place between the blocks that made the project stand out. In fact, it is a very good example of storytelling: even if the table itself did not have an audio narrative, it gave the player a feeling of creating a town with blocks by constantly updating the transportation plans projected on the table immediately. The construction of the town plan follows a clear set of logic, which inspired me to draw out detailed game scenes and logic for my game.
I carried research on technical feasibility for the game, and have explored some technical feasibility of tracking and realization.
PoseNet is a vision model that can be used to estimate the pose of a person in an image or video by estimating where key body joints are.
Pro: Very accessible to the audience, because it is browser-based.
Con: Tracking is a lot less stable/reliable. The game scope is very limited since it mainly supports 2D visuals.
Unity 3D with Keyboard Control
Unity is a real-time development platform. It provides service to create 3D, 2D VR & AR visualizations for Games, Auto, Transportation, Film, Animation, Architecture, Engineering & more.
Pro: accessible to the audience as long as they can download the game and play on PC
Con: not immersive
Unity 3D with Vive HMD + Vive Tracker
Pro: gives a possibility to simulate more natural interactions, like shooting arrows, etc
Con: the Vive Tracker should be used accompanied by Vive Controller, and it only gives position feedback so not much input from the player to engage in a game experience
Unity 3D with Vive HMD + Vive Controller
Pro: SteamVR gives a lot of possibilities to program the feedback from the controller to the user. It is well-supported in development.
Con: Vive can only power the game when HMD and Vive Controller are powered together, otherwise the calculation will get inaccurate. This means that it is difficult to let the player only user controller, not wearing the headset to play the game.
I ended up compromising the more engineering-heavy plan, which requires more engineering to create a headset-free immersive experience with controllers.
The visual design should appeal to children and encourage room for imagination. The overall design for the virtual space should be abstract and low-poly.I was trying to create scenes with Google Tilt Brush. However, it ran into a lot of compatibility issues with the SteamVR library (only compatible with Steam VR library v.1). Having tried both Google Tilt Brush and Google Blocks, I decided that Google Brush is more ideal of volume heavy character and object modeling comparatively, and more compatible aesthetically.
I base the game experience, character appearance based on the research on whales evolution. The journey includes key moments of whales evolution on their body feature, as well as key moments of geological shift that informed their transition from a land mammal to a sea mammal.
I made several prototypes and interaction scripts in C# in Unity.
The game development involves some testing in technical possibility within the constraint of time. It scaled down the scope of the virtual world, as well as the limited realization of moving whales due to lack of time to rigging and character animation in Maya exporting to Unity.
I use the development prototypes, I came up with many key insights into the way of interaction. Including how players navigate the space with teleportation, as well as balancing the amount of audio information with roaming time in the virtual world for the player.
Meanwhile, I got important insights into the interaction when tested on small children with VR. I realized the HMD is to heavy for children between 4 to 7. I also realized that the height difference made the game less reliable for children to experience in VR.