Yale students collaborate on a prize-winning virtual reality film


For her senior project, Yale undergraduate Celine Tien ’17 wanted to create a film that pushed the boundaries in storytelling.

Her new film, “Pippa’s Pan 盼” — believed to be the first hybrid live-action, virtual reality (VR) short film — is still a work in progress, but already it has garnered attention: It won third prize in the AT&T VR/AR Challenge, selected from 67 submissions in the virtual and augmented reality platforms in three rounds of judging. As third-place winners, she and her creative team for “Pippa’s Pan ” won a $5,000 cash prize. During the finals stage, Tien and some of her teammates — the youngest in the competition by at least a decade — were invited to demo the work-in-progress at the 2017 AT&T Developer’s Summit and Consumer Electronics Show in Las Vegas in January.

“Pippa’s Pan 盼” is a story about a woman suffering from Alzheimer’s disease who is trying to recapture memories of her husband. As is typical in VR, the film is “experienced” rather than “viewed” by donning special headgear, which allows the “agent” (the person experiencing the film) to become immersed in its environment, in this case the computer-generated “forest” of Pippa’s mind. The forest is a space of both sights and sounds: trees and flowers in deep reds and purples fill the view, in which a shadowy Pippa moves as she hears her husband’s voice beckoning her back in time. The agent can engage with objects in the film, by picking up a flower, for example, or “transporting” to a different location in the forest. These actions influence the memories Pippa retrieves.

Tien, a film studies major, worked for the past two years at DreamWorks Animation in Los Angeles and Oriental DreamWorks in Shanghai and previously acted in a number of films and television commercials. On the advice of a previous boss, she decided to explore VR as a filmmaking platform, and spent some time last summer visiting major entertainment and technology companies in the United States, Taiwan, South Korea, and in China, where she lived in her youth.

“I see a lot of potential in this new technology and platform,” says Tien of VR. “My senior thesis for film studies is about how VR storytelling deconstructs our conventions of traditional storytelling by allowing the ‘audience’ to engage with the story in a different way. In ‘Pippa’s Pan 盼,’ we created a non-linear story, where the ‘agent’ affects the experience as he or she engages in it. This seemed like an apt way of conveying a story of love and loss — of entering into and holding on to memories.” 

Tien needed a whole crew to help her develop the innovative film, including animators, 3-D modelers, a sound engineer, computer programmers, musicians, and more. She enlisted the help of two friends from Yale — electrical engineering student Julien Soros ’17, who serves as sound engineer for the project, and computer science student Charlie Proctor ’17, an engineer — along with students from the University of Pennsylvania, Brown University, the School of Visual Arts, and the California Institute of the Arts, as well as some professional artists and musicians. Altogether, some 20 individuals — many of whom have had experience working at such entertainment and tech companies as Pixar, Facebook, and Oculus — have been involved in the creation of “Pippa’s Pan 盼,” with much of their collaborating taking place remotely.

Soros, who is majoring in electrical engineering, says that his interest in sound drew him into the “Pippa’s Pan 盼” project. His work experience has included helping build acoustic and electric guitars for Veillette Guitars, and designing and implementing system control algorithms at the futuristic transportation company Hyperloop One.

“Sound effects are really important in VR. If the sound doesn’t line up with what you are seeing, it could feel uncomfortable or be a less immersive experience,” says Soros, who is developing a sound spatialization software algorithm for his own senior project. 

For Tien, one of the most important elements of her project was to revolve her story around a Chinese couple, a reflection of her own background and her desire to see more Asian people in film.

“The underrepresentation of Asians in film was one of my biggest frustrations while working in Hollywood,” says the Yale undergraduate. She hired two Chinese actors from California to play the parts of Pippa and her husband. Their actual movements are incorporated into the built scenery of “Pippa’s Pan 盼” using light field technology and motion-capture. For the music in the film, Tien and Soros are also collaborating with instrumentalist Yu-cheng Lin, who plays the erhu, a traditional Chinese string instrument.

Three young people stand in front of the AT&T logo.

Tien, Soros, and other team members explored a wide number of American films for inspiration and ideas in the early stages of developing “Pippa’s Pan盼,” including the animated films “Up!” and “Inside Out,” and the love story “The Notebook,” among others. They also referenced various paintings and drawings to give them ideas for scenery, texture, and light. Tien took pictures of trees in New Haven to send to teammates working on modeling the forest for the film.

Tien and Soros say they enjoyed watching people’s reactions while demonstrating “Pippa’s Pan 盼” at the AT&T Developers Summit in Las Vegas, where they led many guests through their very first VR experience.

“We were able to demo the film for a broad, diverse group, and we got a lot of feedback that made us go back and rethink how we are telling our story,” says Soros. “It was fun to watch how different people’s personalities came through during the experience. Some just stayed still, took it all in, and appreciated the scenery, while others intuitively knew exactly how to engage in that environment. We had one fearless young girl who enjoyed the VR experience so much that she came out saying she now wants to be an engineer. That was pretty cool!”

Over spring break, Tien travelled in China to meet with investors and potential distributors for “Pippa’s Pan 盼” in that country, where VR is currently more popular than it is in the United States.

“In China last summer, I visited a mall in Shanghai, and there was a VR experience offered on every other floor,” says Tien. “I truly believe that this immersive media is how the future is going to go. I think VR has infinite possibilities, so I think it is really important that younger, diverse content makers and engineers get their hands in this and shape the future of what people experience in VR.”

For Soros, VR is especially fascinating because “it’s an example of technology driving the industry, whereas in a lot of content-creation situations, it’s the content driving the technology,” he says, adding, “In VR, there is so much potential for many amazing engineering feats that really interest me.” 

Tien has founded a VR startup and will continue her work in innovative film after graduation. Before then, however, she, Soros, and the rest of their team are seeing “Pippa’s Pan 盼” through to completion, with the full film expected to be ready in April.

“Creatively, ‘Pippa’s Pan 盼’ has been a huge challenge,” says Tien. “You realize the immensity of what it takes to create a VR film, where there is so much you have to build from nothing. When the film is done, I hope people come away with the story itself — a woman’s love story and her efforts to reclaim her memories even in her last moments. I also hope they will see VR as a meaningful platform for storytelling, one that is not bogged down by Hollywood conventions. What’s special about ‘Pippa’s Pan 盼’ is that it offers a new way to engage with a story.”