Eliza McNitt: Directing the Machine

CHARLIE MELCHER:

Hi, I’m Charlie Melcher, founder of the Future of StoryTelling. Welcome to the FoST podcast. Today’s guest is Eliza McNitt, an Emmy nominated filmmaker and one of the most innovative voices working at the intersection of science, technology, and story. Her work blends the poetic and the technical, and fearlessly explores the boundaries of new media. You may know her from SPHERES, her groundbreaking virtual reality trilogy that invites viewers to journey through the cosmos. It won the Grand Prix at the Venice Film Festival and made history as the first VR experience ever acquired out of the Sundance Film Festival. And if you haven’t seen her most recent work and ANCESTRA, you should definitely check it out on YouTube. It’s a deeply personal and cosmic story of her birth created in collaboration with Google’s DeepMind to bring the camera into unseen places. In today’s conversation, we explore Eliza’s creative evolution from her early experiments in VR, to working with collaborators like Darren Aronofsky and Google’s AI labs to defining a new visual language for storytelling in the age of machine learning. At the heart of it all is her deep fascination with a cosmos of reverence for science and a commitment to making the invisible visible. Please join me in welcoming Eliza McNitt the FoST podcast. Eliza, welcome to the Future of StoryTelling podcast. We’re so happy to have you.

ELIZA MCNITT:

Thank you so much for having me.

CHARLIE MELCHER:

So I’ve been a fan of your work for a very long time, since I first saw SPHERES and actually, I guess it was at Sundance where it premiered, and then again I saw maybe a later version of it at the PHI Center in Montreal. So tell us how you first got involved in virtual reality before we talk about some of your newer work.

ELIZA MCNITT:

Wow, thank you so much. My journey into virtual reality actually started at Pioneer Works, and that is where I experienced VR for the very first time, and I put a VR headset on and it was a awful experience where you are on a rollercoaster and I felt disembodied and I felt —

CHARLIE MELCHER:

Wait a minute, you can’t be — my staff got me a VR headset and the first thing they had me do was a rollercoaster ride. I got sick for the entire day. Did you have that too?

ELIZA MCNITT:

Yes. It was a awful experience. Awful and a terrible entry point into this technology, and yet something about it was so intriguing and enticing and the possibilities felt really endless. The first project that I created was a piece called Fistful of Stars, and that was a journey into the birth life and death of a star. It’s from creating that piece that I received an email from Protozoa, which is Darren Aronofsky’s company, and it was literally like a dream where they just said, “What are you up to next? Come in for a meeting.” I was very, very inspired by this idea that when two black holes collide, they send out a gravitational wave, which is like a song that ripples through the cosmos and eventually reaches us here on earth. And that was the creative launching point for SPHERES. I hope that’s what was so exciting about the project to people is that I think it was the first time that you got to step inside of the universe in a way where you became a character within that story. And that’s so unique for me about virtual reality is you get to embody these forces of nature that are inaccessible to us. It was a remarkable journey for me.

CHARLIE MELCHER:

So tell us about your newest piece, ANCESTRA.

ELIZA MCNITT:

So my next really exciting exploration with new technology is a film called ANCESTRA, which is a really deeply personal story about the day that I was born and it fuses live action and generative AI to tell the story about the power of a mother’s love across the universe and beyond. It’s about my mom. She went into the hospital for a regular checkup when she was nine months pregnant, and they told her, you have to have an emergency C-section right now, and if you don’t, your baby’s going to die, and so will you. So I wanted to recount that story, but through a cosmic lens. Again, it was another one of those dream moments where Darren had this new partnership with Google with his new AI venture called Primordial Soup. They had just created a partnership to create three short films and wanted me to direct the first one. So it was also really exciting to be in a situation where I had full creative license and that is very rare. So it was really a dream project.

CHARLIE MELCHER:

And you used AI in a very original way for this with some original approach to prompting. Tell us about that.

ELIZA MCNITT:

I think what people fear about AI is that there’s this misconception that you just write into the model, write me a script, make me a movie, and the process involves so much more craft than that. And I actually worked on that film with a team of 200 people and 20 dedicated artists to create a film in 10 weeks. And the process only began with text prompting where you type into the model, but then we really had to come up with a much more poetic approach to prompting. And so we went far beyond just text. So for example, I wanted to have this scene of traveling through the mother’s brain, through her body and into her womb to see a c-section from the perspective of the unborn baby. What the team did was create video to video reference where they literally filmed a umbrella on a chair covered in wires and then put it through the model and transformed it into this beautiful nebulous cosmic interior of the human body. And it’s mind blowing that that’s possible.

CHARLIE MELCHER:

I mean, I’m blown away to hear that you’re literally making physical models to create footage to train the AI to be able to see the world in a different way. And you had 200 people working on it. I mean, some people talk about how AI is going to eliminate a team of filmmakers. I think you had more working on this than if you had just shot it in the camera.

ELIZA MCNITT:

Yeah, I don’t know many short films that have hired 200 people, so we felt very proud of that. Yeah, I mean I think that that also was such an important part of my process with Google was defining how as an artist I wanted to use the tools and it was very important to me that the heart of this project was the humans behind it. And so every head of department was represented from a storyboard artist to our VFX supervisor, Aaron Raf. And most importantly, I was really just hell bent on ensuring that my actors were human. And that was something when I went into Google the first time and they showed me all these incredible tests, I was both so blown away, but also terrified at how realistic the human beings were. And that was a moment for me where I stepped back and I said, I absolutely have to work with actors because there’s no way that a machine can capture the human soul. I worked with an incredible actress whose name is Audrey Corsa, and we actually shot in a real, OR which had just been cleaned up for us from the previous surgery, and she laid on that cold metal table for four hours and cried her eyes out and AI can’t do that.

CHARLIE MELCHER:

But you do have a shot in there where it’s a combination of a live actor and an AI character, right?

ELIZA MCNITT:

Yes. So I wanted to find a solution to filming with newborns on set. It’s very unethical to have a baby. They’re not consenting to be there. I wanted to see if it was possible for us to create a baby that has been generated using AI and then insert it into the live action scene. And so we shot my actress literally holding air as if she was clasping a ghost and then created this baby that we put into the scene. But furthermore, what was so unique about that, which really was something that I feel only AI could achieve, was that I wanted to use the images of me as a newborn baby, and I wanted that style to influence the look and feel of the baby, but also the look of the film. And my father passed away a couple years ago, and so I can’t just ask him, dad, what lenses did you use? What camera was a shot on? What film stock did you use? And so we use something called a LoRA, which is a model that you can train to capture a particular style. And so we are able to train the model on that style and aesthetic and apply that to the film, but specifically to the look and feel of the baby, which was me. And so this is a film about my mom and I also wanted my father’s creative DNA to be part of it as well. It really is a love letter to my parents, and I just feel like that touch is something that you just couldn’t achieve otherwise.

CHARLIE MELCHER:

It’s so beautiful, the idea that your father could come back to collaborate with you creatively on this project and also what a nice legacy for you to have his visual aesthetic now in an AI that you could create other images with him or from him in a way.

ELIZA MCNITT:

I’m sure he would have notes, but.

CHARLIE MELCHER:

So it is such a personal film, obviously it’s the story of your own birth, and yet it’s also again, of such a universal film. It is in a sense the story of every birth, every person. And somehow again, you tapped into the universal and connecting in a way to your fascination with space and the infinite. And where does that come from? How is it that you are so fascinated about space?

ELIZA MCNITT:

That’s a good question. I feel like with SPHERES, my father was an aerial marine photographer, so he would go up in helicopters and planes and point his camera down at the ocean. And I have always been so fascinated about pointing my camera up at the stars, and I feel like that’s just something that, I don’t know, he really imparted on me as a child in the dead of winter at four in the morning, he would come and grab me and we would go outside and watch a meteor shower. And so I just feel like that aspect of being connected to the universe was always part of my story. And black holes are so cool. I find the infinite and the existential dread that comes along with it actually incredibly comforting. I think that darkness is so beautiful because there’s nothing else out there and it’s — I don’t know, I’m really, I find it very captivating.

CHARLIE MELCHER:

So now that you’ve worked with AI, are you feeling that this is a tool you’ll use regularly? Obviously there’s still a lot of controversy around AI, particularly in the film industry. Tell me about your take on it now.

ELIZA MCNITT:

Yeah, there’s a lot of resistance and fear about the tools, and that comes down to the anxiety of job replacement and feeling like artists will become irrelevant. And I think that humans are so central to creation and ultimately AI doesn’t have the ability to reflect, and most importantly, AI doesn’t have taste. And so it really is the role of artists to be the central part of this story and to use AI as a tool through which they can create. I do think that more and more, we’re going to be seeing Hollywood and the film industry embracing these tools and the pipelines because for example, like what we did with the baby, it was a solution to a long existing problem in how we get a shot that is super important for this story without having to actually use a newborn. I think there are ethical ways in which we can use AI in order to our stories and also work with a team of 200 people.

CHARLIE MELCHER:

You mentioned ethical ways — a guest we had some months ago talked about ethical prompting and it’s just basically the idea that you wouldn’t ask for the AI to reproduce something in the style of a specific artist. You would do other kinds of prompts, general things for what you’re looking for, but not a direct mimicking of somebody’s style. But I do find it so interesting that in this case you really did use it to mimic someone’s style. It just happened to be your father’s.

ELIZA MCNITT:

Right, most importantly with that is obviously we have — my family has a copyright to those images. You cannot train a LoRA without that permission, so that’s really important. But I do find that really interesting because I of course believe it is unethical to prompt make me an image in the style of Terrance Malik make me an image in the style of David Lynch. But as artists, it’s an interesting thing because art has been inspired from other art for hundreds of years, and when you make a film, you create a mood board based off of the images of other artists and other filmmakers, and that’s how you pitch. That’s how you get something made. That’s how you communicate to your team. So I do find that really fascinating is as we move forward with specifically AI, what is the boundary for people? And I do think that’s why it’s really important for artists to be at the forefront of this conversation, to be able to dictate how the tools should be used and also demand accountability from those companies that are doing that because this is our work.

CHARLIE MELCHER:

Well, I think it made a lot of sense that the Google team came to someone like you who would show up as an artist and create something that was so uniquely human, which is exactly part of the criticism again against AI, that it’s soulless, that it’s machine making things instead of the eye of an artistic human being.

ELIZA MCNITT:

That’s right. I do feel very strongly that creation stems only from the human soul.

CHARLIE MELCHER:

One of the areas that I’m personally very excited about is storytelling that’s immersive so that we can have an embodied experience where we can have some agency, where we can feel truly immersed, where we can have a social component interactions with others. Are you excited about that as well? Do you see possibly AI playing an important role in that evolution from storytelling that’s flat and fixed and linear to stories that are embodied and responsive and immersive?

ELIZA MCNITT:

It is funny you bring that up because I’m actually remembering when I went to Future of StoryTelling in 2016, there was an experience, a virtual reality experience that I did that you put on the headset and you were interacting with someone who was somewhere else in the world and embodying this character and communicating with you. And that was my first time doing something like that and that really deeply freaked me out. But I think those are also the kinds of experiences I like to pay attention to because it’s really fascinating and cutting edge and kind of creepy, but also has so much potential for stories. Actually, I was the president of the immersive jury this year at the Venice Film Festival and had the privilege and honor of getting to experience so many unbelievable projects. I got to do this beautiful project that is by the PHI Center, directed by Craig Quintero and Phoebe Greenberg called Blur. And that project was this really surreal journey and embodiment of being human and questioning that in this very existential journey it brings you on. And it also had this immersive theater component to it and just an incredible use of space. The PHI Center is so amazing at the embodiment of space and your relationship to that. And that was a project that I just was really blown away by and I think is a signal of this very exciting future of art and immersive. So that was a really unique experience.

CHARLIE MELCHER:

So you’re someone who has created across a whole bunch of different new cutting edge technologies. You started as a filmmaker, but here you are probably best known now for work in VR and AI filmmaking. What is it that draws you to being at the cutting edge with new technologies?

ELIZA MCNITT:

I’m a filmmaker and a storyteller, and what I have always been drawn to is what is the story I’m trying to tell outside of the medium and what is the best way to tell that creating an experience about black holes, VR was just a natural fit for that. And so I think I’ve always just been drawn to these new technology and tools as a means of expression and experimentation. I have always been drawn into taking that risk and diving into this creative challenges that are uncharted.

CHARLIE MELCHER:

So one of the downsides of working with cutting edge tech is that it is evolving so fast, and I know as storytellers, you want to create a story that lasts, that’s timeless, that people can enjoy for years and years, yet things get dated so quickly with technology. How have you dealt with that?

ELIZA MCNITT:

It’s very painful for me to see the version of SPHERES on the Meta Quest 3. It really feels like you took an IMAX movie and then put it onto an airplane, but it’s really beautiful and still, I think the story still really stands up for me. I think that even working on SPHERES, we started the project designing the experience for a specific headset, which then was no longer being made by the end of the project. And when that happens, I think it’s really comes down to the story. And that’s always what it’s important is what is this story you’re telling that is timeless that could be told at any point in time? And so I think that’s always where I start no matter what the tech is.

CHARLIE MELCHER:

I couldn’t agree with that more actually, that it’s the great story that’s at the heart of any of these experiences. I had a similar experience years ago when we worked on our first app and it was a app with Al Gore and it was called Our Choice, and it was very innovative for its day and we were very proud of it, and it got lots of attention, won lots of awards, we collaborate with great people. You can’t see it now because the iOS is so many generations later and it doesn’t work. And whereas I also have a lot of history doing books, and you still can read every one of the books we’ve made, so it’s exciting. But there are some downsides or potential downsides of working at the cutting edge.

ELIZA MCNITT:

Yeah, I mean I feel like with SPHERES, it’s — the number one question I get is how can I see that? How can I see that? And I’m always like, I need to bring a headset to you. And it’s actually been really cool to have that opportunity to show so many people virtual reality and to be able to open their eyes to it. And I really enjoyed running around Hollywood with my Oculus Rift and footing it on the faces of so many amazing artists that I admired. That was a real highlight for me.

CHARLIE MELCHER:

So many of these experiments have a difficulty of reaching a large audience. Certainly VR has that limitation of sort of one person, one headset. But I wonder if you have a thought about when you do innovative work with new technologies and it is cutting edge, it doesn’t always have an obvious distribution mechanism or the obvious to get it out to the masses. Has that been a struggle for you? Do you think that’s about change?

ELIZA MCNITT:

If you want to feel amazing, post your AI film on YouTube and then read the comments? I mean, it’s a great question about distribution with virtual reality. That was something that we really had to pioneer every single time we exhibited SPHERES. It was bespoke. And yet I feel like we’ve really reached so many people. The experience was also bought out of Sundance by a company called City Lights who has brought the film and distributed it worldwide. So the beauty of working with these AI tools too is that you can just put it online. I’ve had so many people reach out to me about the film because it’s accessible and you can watch it and it’s like eight minutes long. And so I do think there is some real value just to putting your work online for people to see because people actually do go see it. And that’s amazing.

CHARLIE MELCHER:

Which touches on the economic model, right? You were funded by Google to do that. You didn’t need to have a ticket sale or a purchase,

ELIZA MCNITT:

Right. Yes. I mean, I have been really lucky with a lot of these immersive experiences to get to collaborate with tech companies who are financing the project and the experimentation of it all. And that allows for amazing creative experimentation without expectation of return. But when we distributed SPHERES, there was that expectation of return and it was a seven figure deal out of Sundance, and they believed in us and invested in us. And I believe that they have — I believe that they have made that money back now. So that’s kind of wild. But no funding is hard no matter what you’re doing. And I think particularly right now in just making films, it’s a difficult, strange moment and it is difficult to be an artist and to create things. And I think that that is part of it is just you just got to keep going.

CHARLIE MELCHER:

Are you looking to work in other new technologies or are you going back to your first love of filmmaking?

ELIZA MCNITT:

Yeah, I am really excited about making my debut feature film and exploring how to use AI in the process and incorporating it into the pipeline and seeing how it can be a additive collaborative tool. I think that would be really interesting to explore. But at the heart and soul of this, I’ve spent a lot of time writing with my incredible co-writing partner, Lynn Renee Maxey, who I met because she wrote the first three seasons of The Handmaid’s Tale. And we both share an undying love for sci-fi. So we have been writing a lot of sci-fi films. We met working on a project called Mars 2080, which was a film that we wrote for Ron Howard for Imagine. We got to take my passion for building worlds and brought on a group called The Future Architects to envision a world on Mars in the year 2080 that had been created through the lens, not of capitalism, but of science. What if Carl Sagan had imagined a society on Mars? What would that look like? After building this utopia, I then felt a desire to write a more realistic version of the journey of humans going to Mars. And so that is the film that I’m working on right now.

CHARLIE MELCHER:

Can’t wait. Very fun. Eliza, what would you suggest to people who are storytellers looking to work in innovative ways or with new technologies?

ELIZA MCNITT:

I think the first step is defining your story and the characters and the heart of what you want to say. I think that’s always the most important launching point. And then I think it’s about what is the right medium to tell this in? And what is so incredibly exciting about AI and VO is that anyone can have an idea and start to create it. And I think if I was in film school right now, I would be using AI tools all the time just to crystallize your vision before you even get on set and to experiment and play and be able to have these worlds created before you even go to set and waste everyone’s time because you have never directed before and are going to take an hour to make a decision. So I think it’s such a great tool that I also hope enables and inspires people to tell their really personal stories that nobody was going to finance before. And so I think for the independent artist, I think it’s really empowering to have access to this new frontier.

CHARLIE MELCHER:

Your ability to sort of dream of the stars and then reach for them and then seemingly to accomplish it over and over again is quite remarkable. Eliza, thank you so much for sharing with us on the FoST Podcast. It’s been a pleasure to have you here.

ELIZA MCNITT:

Thank you so much. It’s really an honor and I really appreciate the conversation.

CHARLIE MELCHER:

Once again, I’m Charlie Melcher and this has been the FoST podcast. Thank you to Eliza McNitt for joining me today. Her work is a reminder that we as storytellers and artists have the ability to define the uses and creative possibilities of these new technologies. When we approach these tools with creativity and care, we may unlock entirely new ways of understanding and seeing the world around us. As you may know, in just a few weeks, I have a book coming out called The Future of Storytelling: How Immersive Experiences Are Transforming Our World. It’s available for pre-order now and will be released on November 4th. It’s an exploration in celebration of the coming renaissance in storytelling. If you enjoy this show, the book is for you to learn more about it, you can visit our website at fost.org or pre-order it now, wherever books are sold. The FoST podcast is produced by Melcher Media in collaboration with our talented production partners, Charts & Leisure. I hope to see you again soon for another deep dive into the world of storytelling. Until then, please be safe, stay strong, and story on.