Episode Transcript

Charlie Melcher:

Hi, I'm Charlie Melcher, founder of the Future of Storytelling. Welcome to the FoST Podcast. Working with emerging media as a storyteller requires a special person. Not only do you have to wade into the turbid waters of developing and using nascent technology, but you also have to rethink and reinvent even the most basic elements and pillars of our narrative craft. It requires experimentation, and innovation, and techniques, thought, theory, and even language. This is something our guest today, Jessica Brillhart, knows a thing or two about. Jessica's the director of USC's Institute for Creative Technologies Mixed Reality Lab, and founder of Vrai Pictures, an immersive VR content company. She's considered an expert and pioneer in the field of virtual reality for her early work with a jump platform at Google. During that time, she developed a whole new way to talk about film in the context of VR and devised many of the groundbreaking techniques that are still being used in the world of VR today.

Charlie Melcher:

Jessica is an innovator across many fields of immersive storytelling. As a director, writer, and theorist, she's worked in spatial audio, virtual reality, and now in computer-human interaction. She's collaborated with NASA, Bose, the Philharmonic Orchestra in London, and Google's Artists and Machine Intelligence program. It's truly a pleasure to have such a talented and groundbreaking innovator for immersive storytelling on the show. Please join me in welcoming Jessica Brillhart to the FoST podcast. Jessica Brillhart, welcome to the FoST podcast.

Jessica Brillhart:

Thanks, Charlie. It's good to "be here."

Charlie Melcher:

Yeah, it's great to be here with you. You've been such a brilliant creator of virtual reality. And somebody who has helped to define and push the language of the medium. But at one point you were just starting with it like everybody else and had never used a VR camera before. What was it like for you when you first had to figure out how to use that fancy rig?

Jessica Brillhart:

I wouldn't call it fancy, but it certainly was something. I didn't look to be involved in VR or anything immersive, to be honest. I was at Google at the time. I was a filmmaker there for five years as part of the creative lab, I was getting into a bit of a rut. And, just one day, I got an email in my inbox from a group of engineers who were part of the Cardboard team, which was a very small team at the time. They asked if I was interested in creating immersive experiences and I didn't really know what that meant. And so, I didn't respond because I thought maybe they had sent it to the wrong person. And then I got another email a few days later. So finally I responded and said, "I'm open-minded. I'm happy to check it out."

Jessica Brillhart:

And so, ended up following it to mountain view. And it was their demo day. First time they were showing off some stuff. And so, the engineers put me in a headset and they showed me this thing and it was almost a 3D printed rig circle with 16 GoPros. And, I don't even like working with one GoPro and here I had 16 GoPros in this array. And, I just looked at it and I said, "Okay. Well..." And asked them point blank, "Well, what do you want me to do with this?" And they looked back at me and said, "Well, we were hoping you would tell us." Which started my journey into this crazy space.

Charlie Melcher:

That rig is basically holding 16 GoPros that are facing in all directions. It has a stand on the bottom.

Jessica Brillhart:

The way the rig is actually created, it has the cameras oriented in such a way that you are able to stitch together this footage into one seamless stereoscopic video sphere.

Charlie Melcher:

So you discovered this crazy 16 GoPro rig thing, it's big, it's bulky, it's awkward, it's not like any camera you've ever shot with.

Jessica Brillhart:

Well, you know what? The rig itself was actually pretty portable. Essentially, I was tasked to go out into the world and film with it. And, it was just me and my producer at the time, Nick Cadner. And, we were out there. We went to Puerto Rico. Well, we lost one of our rigs in Puerto Rico, because we were on Arecibo, what used to be the largest radio telescope in the world that no longer exists at all. We were on there and there was this platform that I wanted them to move, but the worker there and us, we had no idea that it was going to move that much. And so, it moved and the rig just went, "Poof".

Charlie Melcher:

Oh no.

Jessica Brillhart:

And all the GoPros went everywhere. So we went to back to San Francisco to get a new rig. Filmed in the redwoods there. Went to Tokyo and a place called Bunny Island. It's an island that is usually filled with bunnies. That is one of probably my lowest points in the entire trip, oddly enough.

Charlie Melcher:

Why is that? Why was that a low point?

Jessica Brillhart:

Ugh. So, we get to this island, which requires you to take two trains, a bus, and a boat. It's an eight hour trip. We get there. This island just has tons of bunnies on it. And, you see all these YouTube videos of these people being attacked by a big crowd of fluffy adorable bunnies.

Charlie Melcher:

Sounds great.

Jessica Brillhart:

And of course it's like, "Yeah, wouldn't that be great if you were in this experience, and just these bunnies just suddenly jumped on you? Wouldn't that be awesome?" So, we get there. I'm sure, if anything else fails, or if everything else fails, the Bunny Island footage is going to be what really sells this thing. We get there. And, the bunnies are nowhere to be found. They're gone. And the ones that we find are fat and sitting under trees or under bushes. And it turns out that the bunnies had been fed a lot, because there was a holiday before we had gotten there. So, everyone had gone onto this island and fed the bunnies loads of stuff. And we came with kibble. So it wasn't like we came with anything that was going to be attractive to a full bunny. These other people had carrots, and leaves, and all sorts of stuff. And we were just very ill-prepared.

Jessica Brillhart:

It was really hot, so the rigs weren't working, because I mean, they're, again, 16 GoPros, they're overheating, and if one overheats the whole thing's not going to work. And I think at one point a bunny does show up and tries to eat whatever kibble I had in my hand. And without even realizing it, I just swatted it out of the way. At that point I was like, "What am I even doing?"

Charlie Melcher:

Down in the dumps on the Bunny Island.

Jessica Brillhart:

It's crazy to think that way. But yeah, it was. I wasn't sure what was going to happen to me at that point or anything, to be honest.

Charlie Melcher:

And the goal of this was to do a around the world trip of footage that could then be turned into a VR film?

Jessica Brillhart:

Precisely. And, when we got back with all of this footage, I had a month to create, essentially, a montage of all of the world experiences that we had. I was banging my head on the desk, literally, because, I mean, nothing was working for us. You couldn't edit 8K footage of this kind in Premiere. It just didn't happen. No one's ever done it before. So again, we were chugging along. And I had to figure out a language for editing VR in, at that point it was, a week. I think, still didn't hadn't figured it out.

Jessica Brillhart:

So, I've been thinking about this the way that a film editor thinks about this, as stringing a bunch of frames together. But what was happening was, my frame wasn't necessarily going to be someone else's frame in the experience. And what I was dealing with were worlds of potential frames that had infinite possibility of what could occur. And my job was really to figure out, "Are there patterns here? Are there attention points here? Where is the frame most likely going to land? Where could it land?" And, that was really the key. It wasn't about me at all anymore, or my perspective. It was about understanding the possibilities that this world that I captured delivered.

Charlie Melcher:

And that opened up a whole new set of thinking for you about what the language of VR was.

Jessica Brillhart:

Totally. I mean, once I understood that I was working with a world-based medium, not a frame-based medium, I was like, "Well, what else is different? What else can I translate here? If the frame is a world now, then how do I determine where frames could fall?" And, I started really thinking about attention and engagement, which I discovered and is really the language for all of this. It's all about attention. It's always been. And I knew that there are all sorts of ways to explore that. Like, "What does it mean to have a close up, medium shot, and wide shot, now that this was all about attention?" Well, you could think of it as if there is a clear one attention point, a single attention point, you're engaging with that probably the most. In a close-up in film, the frame that I show you is on a singular object, usually a person's face, right? And so, I'm paying attention to that person only.

Jessica Brillhart:

So you could see that in some ways translating. So, being able to capture that feeling in VR really helped to solidify that point. It was more about energy, establishing space, allowing for you to get settled, and just to experience the environment around you without any objective. And then, if you take all these translations from single points of attention, couple points of attention, no attention, whatever, each of these blocking of experiences, if you strung them together, became a flow. Narrative flow, what does that mean? What is a beginning, middle, and end to a journey?

Charlie Melcher:

I enjoy hearing this story that you just told, because to me, it's a creation story, right?

Jessica Brillhart:

Yeah. Yeah.

Charlie Melcher:

It's both in that biblical sense, but also in the birth of a medium. You're working with what became the jump camera system rig.

Jessica Brillhart:

Right.

Charlie Melcher:

And, that was the first VR film made for Google, made with Google's platform and technology. And, you went on to be the principal VR filmmaker for Google. And as such, had a lot of influence. There were a lot of people following in your footsteps, or being inspired, or using those tools, and getting really excited. And you were, through trial and error, learning and figuring out the language of this medium, of VR, that's distinct from film. And, had the ability to share that through your public speaking, teaching, lectures, and just the good work you were doing. I guess I'm curious, as somebody who was there in the beginning of VR and helped to push it forward, how are you feeling about it as a medium today and the progress that it's made?

Jessica Brillhart:

That's a really good question. I will have to say, I stand on the shoulders of giants. Janet Murray, Jaron Lanier, Scott Fisher, a number of folks who were in this well before I came along. That said, I do feel like a bit of an old foggy at this point. It hasn't been that long, but I feel like I've gone through, I don't know, the trenches and it's such a different world now than it was when I started, for sure. I feel like, VR still has a great deal of potential. I do think there's a lot more to unravel with it. There are things that I'm still trying to grapple with myself.

Jessica Brillhart:

Nick actually passed away. Gosh, I don't even know how many years ago, four years ago. And he was the only other person who really understood what we had gone through. It was awful. He was the only other person who saw what we had to do and went through that with us. And, I had all this VR footage of us together. I would be going and even being in world tour again, I knew he was around there somewhere. And I remember having a conversation with one of his really good friends, who at the time worked at John. And he was like, "I know you have all this footage. Do you think you could send us some for his parents?" And I had a bit of a crisis with that, because I'm like, "Well, I don't know..." I always think about it like, "What if they want him back so badly that they don't get out of the headset?" I know it was a bit of an exaggeration. They probably wouldn't know when to get out.

Jessica Brillhart:

But things like that, where it's like, should you remember everything, is that really what we mean memories, right? The ability to go back in time and be with loved ones, be in places that we visited. There's something really beautiful about that, but there's also something that could be quite psychologically damaging, I think. Especially given the rise of addiction to... For instance, the reason why people are addicted to video games, isn't because video games are violent and people love violence, it's because the video games provide a world that is better than their own. It's like, if I can be on Fornite, and be the champion, and when I'm out of that, I work at the grocery store, and no one pays attention to me, which world would you choose? Right? And so, I think about that when it comes to VR as well, if we create these really amazing worlds for folks to go to, will they ever want to go back to their physical ones?

Charlie Melcher:

I certainly haven't had that problem in VR.

Jessica Brillhart:

Not yet.

Charlie Melcher:

I have to say.

Jessica Brillhart:

Yeah.

Charlie Melcher:

There have been some good things. But I'll tell you, I was just recently visiting the Tribeca. I forget what they call it, but-

Jessica Brillhart:

I guess Storyscapes or something.

Charlie Melcher:

... Storyscapes. Thank you. Yes.

Jessica Brillhart:

Yeah.

Charlie Melcher:

And, looked at a bunch of the VR. And, I have to say, I was surprised that it hadn't evolved more in the many years that we have both been working and experiencing these pieces. And certainly, you're right, there have been previous waves of VR that go back to whatever it was, the late 50s or 60s, and then others along the way. But in this current incarnation, which let's say, got really launched with Oculus, with Facebook purchasing Oculus. And there was a lot of innovation in the early days with people like yourself, and Felix, and Paul, and Chris Milk and others. And then, I feel like, not much has happened in the last five years.

Jessica Brillhart:

So, I have a theory, Charlie on why that is. I just don't think we're good at experiencing anything anymore. This is a medium that's entirely about how we experience the world. It's by nature that. I mean, we're put in place, and then it's all about us interacting with that place, or with that space, or no matter what it is, we still have those same urges to be aware and to be safe. We still have that curiosity of interacting with objects and having those objects respond to us. We just are so used to, I think, screen-based technologies and media that even as a creator, it's hard to grasp the fact that this is nothing to do with you sitting back and just witnessing what's going on. It's about experience.

Charlie Melcher:

I'm all for experience. And by the way, I think that we are still as human beings, as our species, we're designed for experience, right?

Jessica Brillhart:

Yeah.

Charlie Melcher:

We have all these sensors, and senses, and we operate in a three dimensional world. I agree with you that most of us are trained to be passive consumers of our content, sit and watch television, or watch movies, or then the internet, and the world of video games, and web-based games came, and all of a sudden we realized we had agency in a digital world with our content. And, maybe the truth is that it's just that a lot of the VR work hasn't taken full advantage of enabling the person having the experience to really have embodied agency and co-authorship. Some of it is still quite passive, where I'm not a personally a big believer that a 360 view or spheric's view is enough to create a truly immersive experience. I mean, yes, you're surrounded by it, but it's still basically a passive experience. The only agents you have is choosing where to look.

Jessica Brillhart:

Mm-hmm. Yeah.

Charlie Melcher:

Bigger screen. I mean, maybe it's just that the creators aren't pushing the medium far enough yet to really let us have these incredibly embodied experiences. I mean, I think back to the work that DV group was doing with Alice VR or Jack, do you remember that Jack VR. Those were great examples of being in a VR experience, but having a really multi-sensorial and participatory ride of a story experience, those were exciting.

Jessica Brillhart:

To your point, if I'm going to put a human being in this space, I do need to figure out how human beings think. I need to think about how human beings interact with the world. It's human-centered design at its very core. It is literally that. And so, on top of that, beyond just thinking about the human in the loop, you also have to think about the environment around that person and the circumstances under which that person is there. But the hard part, which is the part that I think we could totally do more prototypes about and make it fun, it doesn't have to be something super serious or take forever, is this idea of, "How do I make this world have a better relationship with this person? How do I have the world need that person in order to be what it needs to be?"

Jessica Brillhart:

And right now, the VR experiences that I'm seeing, and I'm sure you're seeing, have a world that it doesn't matter if you're there or not really. You could be somebody else and it wouldn't mean anything. So, if I'm going to bring this person in, I sure as heck have to make sure that the world makes them feel like that. So to your point that it goes into ideas around contextually adaptive assets, like, "Who are you? What's your height? Have you used VR before or not?" That kind of thing. It's something that changes... It's very fluid. It's something that should change all the time. It should never be static. And, the great thing is, people try to replicate how things work in the real world in VR. And it's like, "Why?" You can build whatever you want. You can change everything. You can change concepts of time and space in ways that you can't do in the real world.

Charlie Melcher:

It's an exciting suggestion about dynamic, and responsive worlds, and characters in our stories. And certainly, it's one of the promises of digital storytelling of the possibilities today.

Jessica Brillhart:

Totally.

Charlie Melcher:

And, I think many creators are limited by how things were made before and what they grew up experiencing. And it's hard to sometimes let go, and get out ahead of where things have been. I think you did make the point also that similarly, the people who are consuming these experiences have to have a familiarity, have to be able to be comfortable with, let's say, being a character that's evolving and changing.

Jessica Brillhart:

I agree with you. Making VR is very hard. I don't want to knock anyone for trying at all. Just making this stuff is time consuming. It's not a walk in the park, let's say. On that front, I remember, when I was first starting out, I thought about how hard it was too. It takes forever for me to make these things, and release them, and all this stuff, really frustrated. So, I decided to make some limitations for myself. I spent one day, I only gave myself a day, and I told myself, "You can only use what's on YouTube. And you can't overthink it. No overthinking. Just make something that is weird, and wonderful, or whatever."

Jessica Brillhart:

And so, I ended up taking four panels of the weather channel that were screenshot during different times, during the 90s, I put each panel in... One of the coordinates, there was black space with weather channel panels. And when you looked down it said, "Welcome to Omaha." Because these are all Omaha captures. I don't know why. And then if you look up, you see a gif of Kenny G playing his saxophone. And, the entire time you're listening to Kenny G as you look at these panels.

Jessica Brillhart:

And I didn't think anything of it, I released it. And it took off completely on its own. Mostly because people were like, "It didn't take itself too seriously. You could see it at a bar." And no one's like, "No, no, no, you have to see it this way." No, you pull up your phone, you do it, you could lay on your back and watch Kenny G for as long as you'd like. I had people over the world who were... I had a VR creator from Iran who said, "I don't know what this is, but I really like it." I had someone from Argentina tell me that she remembers traveling to hotel rooms with her parents in the U.S. And, that was one of the tried and true channels that was always on. And so, it was a nostalgic thing for her. She felt safe in it. So, it was a pretty awesome on-ramp for folks to think about 360 or think about these spaces they could create.

Charlie Melcher:

So, you are now thinking a lot about interfaces.

Jessica Brillhart:

I am.

Charlie Melcher:

Is that right?

Jessica Brillhart:

I am. Yes.

Charlie Melcher:

And I guess it's a topic that's always been in your mind, because you've always been interested in technology, and how we interact with it, and humanize it. But, tell me a little bit about your work on interface design.

Jessica Brillhart:

Yeah. I promise I'm not going to talk about like, "I was born in 1984." I do want to mention something about my childhood though. I grew up in a small town in Pennsylvania. My dad was an amateur programmer. He worked mostly in HyperCard at the time. And he used to build programs for me to learn. And so, very early on, I was using a computer, almost like a companion to learn more about the world. And the engineer was my dad. So I knew where it was coming from. It was fairly transparent for me as a kid. And, another great thing about my dad is he also comes from a long line of farmers. So, the natural world was always something that we enjoyed. I had both things happening at the same time, and I never saw them as being mutually exclusive. It was all part of my existence.

Jessica Brillhart:

So, over time, again, I had that benefit of having that duality... Not even duality, it was a singular thing, but obviously two different worlds were emerging. And it was really hard for a while, because as a result, I would have this creative storytelling side, which inevitably led me to being a filmmaker. And then I also had this technical side. So I was living in this duality for forever. Google, same thing. Here I am, being a filmmaker, trying to get close to that kernel of where storytelling and technology can actually meet. And how I could show how these interfaces were human. How do we humanize this stuff? So for me, we're at this point, to your point, where I think, we're building a lot of things in silos, and you're seeing a lot of plateauing happening.

Jessica Brillhart:

So, VR has its thing going on, AR has its thing going on, spatial audio has this thing going on. But, what I can say is that the infrastructure of, say, blockchain is phenomenal. And as it relates to privacy and how we transfer our information to others, including companies like Google, and Apple, and Facebook, or whatever. The control factor, the privacy of the transparency is fantastic. And even the idea of tokenization, NFTs aside, the tokenization in DOWs, all of that is really fascinating and I think play a part. So I can't help but want to put them all together, to optimize that in such a way that...

Jessica Brillhart:

We're going towards this era of spatial computing, there's no doubt about that. We're going to start to make our world our computer, but that means that we need to reconnect with the world. We need to build these systems with the world in mind. And, that is not going to be folders, windows, tabs, keyboards, mice. We can no longer compartmentalize in that way. We don't have the time or the attention span to do that. We're going to be dealing with so much information. So for me, I just feel like this is exactly where my brain should be right now, which is, how do we create an interface? What is that interface that brings together these three domains? Again, the way that it should probably should have been in the beginning. Us, we human beings, the system itself, and the real world, but it also requires us to develop systems and system interfaces that respect the fact that we have that relationship with the world and aims to augment us to thrive in the world in a way that helps us, assists us, in living our lives. No pressure.

Charlie Melcher:

That'll be easy.

Jessica Brillhart:

Yeah, I'll do it tomorrow. Fine. Yeah.

Charlie Melcher:

You got it. Well, I love it just because it does feel like you're working towards something that's going to ultimately be more human, and more natural, and something that will be bringing those parts of the life that you described growing up together. Being able to be real people in a real world.

Jessica Brillhart:

I know I said in the beginning, or wherever, this cut into it, I said, "I didn't look for VR, VR found me." Looking back on that comment, I don't know if that was necessarily true entirely. I feel like, we all put a pin in something that we think is our life's work, or I don't know if legacy is really the right word, but there is something for us out there. And I think, whether or not we see it or not, or consciously understand it or not, I think we do put a pin in it.

Jessica Brillhart:

Charlie, I'm sure you did that too. You're like, "Okay. I don't know exactly where all this is going, but I know if I keep following it's going to take me where I need to go." And that has been my career in a nutshell, where I think at one point, my husband was like, "How do all of these projects actually work together? They're all over the place." I'm like, "You have to trust me that eventually all this, it's all going to collide. It's all going to become cohesive. I can't not do five different projects at the same time. I have to be aware of what's going on, because eventually these things have to talk to each other. And if I don't know or respect one thing, then the whole thing's not going to work. It's not going to function. It has to be something that I consider from all angles. Much like any experience, I guess."

Charlie Melcher:

What was the name of the first piece you did, VR?

Jessica Brillhart:

World Tour. Very, very specific.

Charlie Melcher:

World Tour. Well, I'm excited to join you on your world tour, whichever direction this next phase of your journey takes you, let's go.

Jessica Brillhart:

Yeah, totally.

Charlie Melcher:

Great to be with you. Thank you for being here.

Jessica Brillhart:

Thanks for having me, Charlie. It's always good to talk to you.

Charlie Melcher:

Warm thank you to Jessica Brillhart for joining me on today's episode. You can find out more about Jessica's work, including links to World Tour, Vrai Pictures, and the USC's Institute for Creative Technologies Mixed Reality Lab in the episode's description. My sincere thanks to all of you who listen to our show. If you enjoyed the podcast and want more FoST in your life, please leave us a review wherever you get your podcasts and sign up for our free monthly newsletter at F-O-S-T.org. The future of storytelling podcast is produced by Melcher Media in collaboration with our talented production partner, Charts & Leisure. I hope we'll see you again soon for another deep dive into the world of storytelling. Until then, please be safe, stay strong, and story on.