Derek Ham: Living History in VR

Charlie Melcher:

Hi, I’m Charlie Melcher, founder of The Future of StoryTelling, and I’m delighted to have you with me for this episode of the FoST Podcast. Our guest today, Dr. Derek Ham, is an award-winning VR director and head of the Media Arts Design and Technology Department at North Carolina State University’s College of Design. I first encountered his work back in 2018 when he submitted his VR project, I Am A Man for the FoST Prize. The piece which is about the 1968 Memphis Sanitation workers strike and the assassination of Dr. Martin Luther King Jr. Which took place there in Memphis is one I will never forget. It let me embody the experience of a Black sanitation worker allowing me to see and feel the emotions of that day from a whole new perspective. I’m not the only one who was profoundly moved by the piece. It was an official selection at VR and film festivals across the country, and we were honored to award it that year’s FoST prize.

 

Since then, Derek has continued to use VR to shed light on stories from African American history. His most recent piece, Barnstormers: Determined to Win, gives a glimpse into the experience of African American baseball players in the 1940s by putting us in the cleats of one of the players themselves. And again, his empathetic StoryTelling has made an impact. Barnstormers has been covered by outlets such as NPR and USA Today and was selected to be shown at the prestigious Cannes Film Festival last year. It’s a pleasure to reconnect with Derek today to talk about his work as a creator and educator. Please join me in welcoming Derek Ham. Derek, welcome to the Future of StoryTelling podcast. It’s really a pleasure to have you here today.

Derek Ham:

I’m so glad to be here. Thank you for having me.

Charlie Melcher:

We first met when you had submitted your VR piece I Am A Man to the Future of StoryTelling Prize and you won it and we were super honored to have you come and receive that award. Tell me a little bit about your path. What did you study and how’d you come to VR?

Derek Ham:

I have two degrees in architecture and a PhD in something called Design Computation. When I started my doctoral work, it was two years in where I was doing research into, what I call today, spatialized media, how do we get content to come off the screen? Using projectors, using anything that I could to make content pop off into a third dimension. And as someone with a background in architecture, you’re always thinking more spatial. And so I wasn’t even looking for traditional VR, because even though I had known about it was just too expensive. I knew that price point wasn’t what I was looking for. So I was looking at all these cheaper hacks and other ways to accomplish this stuff. And that’s when I saw this kickstarter project called Oculus. And I knew right then and there we got to get one of those. And I was working with a lab and they were like, “Yeah, get one, order one, let’s see what we could do with it.” And when I got that headset and started immediately out the box opening into Unity Game Engine and started working with it changed how I just thought about what the potential we could do to tell stories, to make spaces come to life, to transport us to places. So I still think of those early days as just being magical in a way.

Charlie Melcher:

I love that you started from a perspective of three-dimensional or spatial story telling. It’s one of the things that I also have been so fascinated by through the journey of Future of StoryTelling, which was to watch the world of story telling, the world of media and communications go from flat two-dimensional screens or pages to this world of three-dimensional where you could step into the story. I think that is literally one of the shifts taking place in the 21st century, is going from the flatland of the printed page and the confines of a rectangle on a flat screen into three-dimensional worlds that we have agency in and can step in and be embodied in and all those things. So it’s not surprising to me to learn that you started with degrees in architecture and thinking about three-dimensional world building, but it’s just something very exciting to me to discover that we share that fascination. And of course then not surprising that you found yourself enamored with virtual reality, which is a sort of about three-dimensional world building and putting people inside something, giving them that sense of presence in that space. Now you were pretty much self-taught though when it came to VR, right? You got that headset-

Derek Ham:

Yeah.

Charlie Melcher:

… And then what happened?

Derek Ham:

I think in a lot of ways it was good that I didn’t have the baggage. I didn’t have the baggage of game development, I didn’t have the baggage of film to give me these rules to say, this is how you’re going to have to do it. So when I talked to other friends and colleagues who were coming from places like film for instance, struggling with, “Well, how do I make cuts and how do I direct the individual to look here, to look there?” For me, architecture, when you put a building into the world, you direct spatially, you don’t control every single way they look, you really are creating a set of rooms, a set of spaces. And so I was used to the vocabulary of spatial driven experiences already, and I didn’t have this baggage that to say, “Oh, I need to tell a story by cuts and edits and close angles and stuff.” I was like, “No, I tell stories of spaces.” The largest hurdle for me when I started getting into it was on the animation side. It’s one thing to tell stories of spaces, but then you start wanting to deal with human figures and avatars and people and it’s like, whoa, okay, back to school here, let’s figure this out.

Charlie Melcher:

Did you see other work at that time? Were there things that were influences?

Derek Ham:

There was a piece, a Pearl Harbor VR experience that I remember an early one that fascinated me with the idea of history, digging deep into history. And I remember seeing that piece and it was just like, again, these were early pieces that was done by Time Life. And I remember these early pieces, they’re making the rules. There’s no conventions of what to do. Should something glow to tell you when to grab it, or do you use a narrator? Do you use music? And so all these things were so experimental, but I remember testing that piece out and saying, “Oh my goodness, like aha.” This is only the beginning for us to start thinking about where we revisit historical moments to ask you to reflect deeply about what it’s like to be there.

 

I was fortuitous in that when I started working on it, Oculus had just started to release a new input device, and that was the touch controllers. And as soon as I heard that they were going to release these new controllers that gave you the simulation of hands, immediately began thinking about, oh my goodness, transporting people, transporting them into the ethnicity of someone else, transporting them into the gender of something else that’s going to be radically different. And for me, it was just perfect timing that those controllers were coming out just when I was starting to work on, I Am A Man.

Charlie Melcher:

So describe, I Am A Man, please for our listeners.

Derek Ham:

So I Am A Man is a virtualized experience that takes you into the Memphis sanitation workers’ strike. And that’s this moment in time where most people just remember, oh, Dr. King was assassinated in Memphis in 1968, but they don’t often know the larger narrative that he was there for the poor people’s campaign, he was there to push against that inequality on wages for working class, and in this case the sanitation workers.

 

And so when I began thinking about this project and I put it together, I wanted to teleport you into a sanitation worker and using the touch controllers, the first thing that happens when you look down, you have the hands that of a Black man. And that is transformative for a lot of people. They’re like, “I am out of my skin, I’m in someone else’s lived experience.” And this project, only 14, 15 minutes long will take you through these different stages of understanding some of the struggle that was there right into that very eerie moment. Being at the Lorraine Motel, you see Dr. King on the balcony, and you know something is about to happen. And so really putting you in that place in time, dealing with the gravity of how heavy it was for them to be struggling, the loss of Dr. King, the fight for equality and letting you kind of live that a little bit, I will say it was as emotional for me, building it in a lot of ways as it has been for the people I’ve seen experience it.

Charlie Melcher:

Well, I remember being incredibly moved by the piece, obviously trying to experience it from the perspective of a Black man, which is not an perspective I’ve ever had. And for me, it was a beautifully told story because I felt that it really used the medium in a very unique way to create a stronger emotional response. So not only do you go through this first really associating with the strikers, you can hold a placard that says, I Am A Man. I think I even remember being able to empty some trash, be a sanitation worker first, then be a striker, but then at the end when this peaceful demonstration has turned to a nighttime riot because of the assassination of Dr. Martin Luther King, and you feel traumatic and emotionally upsetting moment because you interjected into the animated world, real footage like documentary footage that we could watch on TV screens through the store window. And then the next thing that happens is this police car pulls up, these police officers get out point guns at you and tell you to put your hands up.

 

And you’re at that moment where you’re sharing that anger that you’re filled with emotion and you’re put in a sort of moral quandary. Do I raise my hands and obey or do I give them the finger? What am I going to do right now? What role am I willing to play here? And I personally had never been arrested at gunpoint by police, so when I raised my hands up, begrudgingly, it created a whole set of emotions in my body that I wouldn’t have had if I were just watching it. They were emphasized by the fact that I was doing it.

Derek Ham:

Right.

Charlie Melcher:

And I just thought, this is brilliant. This is using this medium to elicit a different kind of emotion than one that’s meant to be passively observed. This one is meant to be acted.

Derek Ham:

It’s interesting. You’re not the first person to comment on that moment in the experience, but it’s amazing what the emotional impact can do to even how you perceive the space. So many people have told me it’s like, “Yo, that moment when the police have their guns pointed at me, they tell me to put their hands up.” And I always just kind of sit and listen to that because the reality is, there is no police policeman and there is no gun. It’s a bright light in a cop car and the voice, but it’s almost like a form of editing. When you see in the old films and you hear a car crash and you see someone there and you’re like, oh, that person was hit by a car. You’re like, “Well, we never saw that footage.” Your mind put that together.

 

I do this at the Lorraine Motel, often people they respond of, “Yeah, I saw Dr. King get assassinated.” It’s like, well, actually you didn’t. It fades to black and you hear the gunshot, but that’s the type of spatial and VR editing that happens where we can rely on the person experience it to fill in the gaps sometimes, and then we can just highlight those key moments very intently and it’s amazing what that mind will just fill in the gaps of, I experienced it that way.

Charlie Melcher:

Wow. I feel like I can see the police in the… But there were no police, that’s crazy. That is really crazy.

Derek Ham:

It was audio that was audio and a bright light from the car and your mind will put the pieces together, yeah.

Charlie Melcher:

Wow. Tell us also about using VR to bring history to life. You’ve also done that in your newer piece, Barnstorming, you’ve again chosen an important part of the African American historical experience in America. Why are you choosing this medium to help us experience history?

Derek Ham:

For me, there’s something special about revisiting history through spaces. I’m from Virginia. I was raised in Hampton Roads, which is about 30 minutes out from Williamsburg, Virginia. And even as a child, my mother is retired now, but she was a history teacher in elementary school and so she would famously grab my brothers and I drag us to Williamsburg in summertime and I remember learning about history through spaces. I remember going to DC and seeing things and seeing places and spaces and so for me, every time that I think about those really meaningful reflective moments in time where I’ve thought about something in history and I actually went there physically, it’s not just about a photograph, it’s about a spatial kind of, I’m sharing the world, the air, the space that these people inhabited once upon a time. And that’s what I find so special about virtual reality, it better recreates that space for you to be reflective about the historical moment.

 

And so I have been really drawn to finding historical moments that are partially known, but there are some facts about it that have been left out. And so for I Am A Man, for instance, we know so much about the highlight bullet points, Dr. King assassinated in Memphis, but it’s like, wait a second, there’s all these sanitation workers and they fought for equality and a few months later they got a major bill ratified for their city and there’s some components to that, that moment in time of Dr. King’s assassination was so big, it made the other key moments kind of disappear from the brightness.

 

So when I look at Negro League baseball, for instance, you look Jackie Robinson, first player integrates baseball, and you say, wait a second, let’s tone that light down and look at more of the story and then you see, wait a second, there is this guy named Larry Doby, who two months later in the same year also integrates into Major League baseball. And then you start digging, you start seeing all this richer aspect of the history. And so for me, I love those moments where we can find those parts of the historical experience where it’s like, I think I thought I knew the story, it’s like, no, you only knew the tip of the iceberg. How can we shed light on so much more that was happening at that time?

Charlie Melcher:

As you describe it, I think about the tradition of good writing, and often it’s the specificity, it’s bringing up some very small details that somehow make people often sort of feel like they’re there when you’re reading a novel, for example. And I think about now the spaces that you create in your VR pieces, the ballpark, which is extraordinary or several ballparks and really feeling something about the time and the place through being in that space. But I also think about how you use action and gesture as well. I mean, in Barnstorming, this more recent VR piece, you let us sort of catch a ball and then take it out of the mitt and throw it and that’s such a natural gesture. I mean grew, I wasn’t a serious baseball player, but certainly grew up playing catch as a kid, and I did it with my son, it’s still something we enjoyed doing together. It’s such a familiar action, I just feel like you’re also building a language of physical actions that is helping us feel authentic in that place and time in that space. In the same way a great novelist might come up with a detail about clothing or somebody’s physical appearance or something.

Derek Ham:

What I was trying to do here is bring you into the community. Those players, it’s like a fraternity and I’m like, “Hey, step on the field into fraternity, play ball, hear them laughing in the background, just hanging out together.” Even before that, a scene of being in the sandlot, I was trying to bring people into these intimate spaces where you’ve probably, again only seen a photograph and to say, “Wait a second, this isn’t a woe is me moment, this is hey, we’re going to make the most of this and have a blast and do it at the highest level.” And being able to put you into the smaller, everyday mundane moments like a practice or a sandlot, I think is just as important to really visualize what it was like to be in this time to see these players instead of just the big hero moments of the black and white of someone hitting that home run. And I give you that too, but I wanted to make sure that those subtle moments are there.

Charlie Melcher:

But clearly you also have been using the medium of VR to help give people some insight into racism and prejudice in our country. And it’s deep tradition, sadly. I wonder if you could talk a little bit about that and what your hope is there to use VR for that purpose?

Derek Ham:

Yeah, you look at a moment in time, and what we often forget is whenever we review a history, we see a documentary, look a photograph, you’re looking at it from the lens of your own eyes and so it’s hard to sometimes pause and go beyond that. And I think that’s that true empathy is like, “Oh, that was hard for those people.” Imagine yourself being that person and what I was trying to do here is position you, and especially with Barnstormers and kind of shed just a little bit of light on that strangeness that America has always dealt with, with the love and appreciation for certain cultural aspects, and then the shunning away when you move into the neighborhood.

Charlie Melcher:

Right, was there a little bit of that for you in terms of trying to imagine what it was like back then?

Derek Ham:

Absolutely. Barnstormers going, if I Am A Man, takes you to 1968 by the time I get to Barnstormers, going even further back. So now we’re talking 1947, 1946, and it’s like I’m going further back in time with these pieces, but with those, a lot of historical research. I went to the Negro League baseball museum in Kansas City, the director there has been very supportive of this project and was very instrumental in giving me some key notes of things that this piece had to have. And then just so much time just reading and pouring through and listening to interviews and really getting a full understanding of this era in time, but also these men at the time. And that’s why I discovered and really started digging deep into this idea of the celebration of this aspect and really started questioning, if we talk about empathy as being this thing to feel someone else’s emotions and to feel and to connect with them, why does it always have to be the pain and suffering? The hero moments can be just as empathetic too.

 

And so again, you go back into African American history, people want to say empathy, empathy, oh, I want to share the suffering. I’m like, “No, can we share in the hero moments too?” Can you be just as proud that these guys are doing this and hitting these home runs? And I knew for any African Americans who try this piece on, they instantly start to feel hype and clap and they’re getting all behind it. So I’m like, culturally, does that transfer over? If you’re not Black, are you putting the headset on and just super excited to hear about Josh Gibson, Satchel Paige and see how successful they were. That’s what I was really teasing out, trying to say, let’s look at the full spectrum, let’s understand people’s pain, let’s understand their successes. Let’s feel sad when they need them to be sad with them and let’s rejoice and be happy with them at those moments as well.

Charlie Melcher:

That’s beautiful, yeah, it’s a great point that empathy doesn’t just have to be around suffering. Empathy can be around success and joy too. As we said earlier, you are a self-taught VR creator and now you’re an educator and you teach others, you run a program to help with this type of story telling. How important is your own creative process in informing your teaching?

Derek Ham:

One of the things I tell students is not to be limited and to do work in silos. I tell them all the time, bring every aspect of yourself to your projects, you never know what your inspiration might be. One of the things that I did, I’ve done in both the VR pieces, I’ve dealt with music in a very strong way. And sometimes people ask me, it’s like, “Well, how are you making these music decisions?” And I have to tell them, well, when I was an undergrad, I sang in a classical choir, I sang in a gospel choir, I have this music background, I can go even back and say, I played piano and trumpet when I was in high school, even further back.

 

And so oftentimes you go down a road and you say, well, that’s music and we push that over there, I’m in architecture now, and you go into something like VR, and I was just pulling from all of my background. I’m pulling from all of my understanding. And so I tell students from the very beginning, bring all of you. Do you skateboard? Do you surf? Do you play basketball? Do you sing? Do you do cosplay? You playing Dungeons and Dragons? I don’t care. All of those aspects can be brought to the table when it comes to a creative endeavor. And I start there and I let them know that they can bring them whole selves to the table when they do a design project, and especially something with VR.

Charlie Melcher:

Such a good note. And it resonates for me personally. I was one of these wacky people who created his own major in college. They called it a special divisional major and I put together literature and photography and film a little philosophy, and we called that a major. And at the time, I remember my dad saying like, “Wait, what are you majoring in?” And of course, what are you going to do with that? And then lo and behold, it’s exactly, it’s so relevant to what I ended up doing with my life and now it seems like it made perfect sense, but then it didn’t fit in any neat box. So tell me a little bit about how you are thinking about designing this program that you run now?

Derek Ham:

So the program at North Carolina State, it’s called Media Arts, Design and Technology and you look at a name like that and it’s already implies that’s going to be a mixture of so many different things. And for me, at the core of it is about story telling. It’s like, all right, what type of stories do we want to tell? What kind of mediums? What kind of genres? But then where are those boundaries and how can we blur those boundaries? Is something animated? Is something a picture? Is it a film? Is it experience? Do I have agency to interact and change things? And so we’re living in this beautiful age where I think we’re going to see new genres of content pop up. And so the challenging thing often is when you have a young high school student, they want to come into college and they have this from their perspective, “I know what I want to do and it’s that thing.” And they can point at it and you try to pull them back to say, “Hey, just because you know it’s that thing…” I want to pull and break everything apart. That’s a design process, let’s break it all apart to its pieces of components and then rebuild it into see what we find. And that’s where you find the innovation in new types of products and things.

Charlie Melcher:

Derek, what’s exciting you? What are you seeing? Either through students or other forms out there, where are those inspirations for people who are inventing new forms that are turning you on?

Derek Ham:

It’s the new genre of media. I love everything. I love TikTok. I love the idea that someone’s figured out like, oh, we could watch things at short spans. I love augmented reality. I love the idea that in front of our eyes there are new genres of entertainment. And it’s not kind of like a big announcement, someone like announces on the internet, this is the new genre, it just organically happens. So you look at something like people now sitting in their houses, the television on is on in the background, but they’re being entertained with their phone in their hand. And it’s like, whoa, could you have predicted that more eyes would be on TikTok videos and small Instagram feeds and things like that, as opposed to the television, which had been the holy grail of entertainment for so long?

 

I’m so excited that we’re starting to see the birth of new genres of story telling and new genres of media. And so with AR and XR, they’re hungry, they’re nipping at the button. The one thing we have to do is resist the temptation to simply classify it as something from the old. VR is just a gaming device, or VR is just that. I really hope and that’s where I push the students to say, be bold. What’s the new, what is the new genre? What’s the new thing that people will walk away saying, “I’ve never experienced something like that. I want more of that.” And that’s what excites me, this idea that we’re at the birthplace of so many new genres of story telling and of entertainment.

Charlie Melcher:

Are there other things that you encourage your students to do or to think about in the program, other ways to get them to think originally?

Derek Ham:

I’ll say even with the kind of, the freakout, a moment of AI, that was the kneejerk response that so many people had, even some of the students, the kneejerk, it’s like, “Oh, what’s this about? It’s going to take away my creative job and this…” I’ve been telling students, it’s like, you can’t afford to put your head down. You have to keep your eyes open to every single thing. You have to see, hey, how can I reappropriate this for my gain, for my good? And so right now, that’s been my big encouragement, is to keep pushing students to say, if this new thing is out, have no fear. Investigate it, see what that’s all about. Don’t make your mind up about it too soon. And so I think every single time something bold comes forth that’s radical, that’s disruptive as we are now dealing with AI, I tell the students, jump in, test it, break it apart, push it to its limit, see where it’s going to be and be bold about it because we’re going to have it, let’s all use it in the way we want to use it.

Charlie Melcher:

Is there something new that you’re working on that you might mention or share?

Derek Ham:

I think for me it’s more of the type of experience. I said, I’m like, I’m interested in this genres exploring and I’m very interested in more shared experiences. I love VR, I’m proud of the work that I’ve done over the last several years, but so many of those experiences are still an individual and a headset and there are lots of technical challenges to make something like I Am A Man or Barnstormers with multiple people in the headset. So I’ve really have been spending so much time investigating more communal things, technologies that allow people to sit around a table together, technology that allows people to sit around campfires together. Where does technology bring the huddle moments for a shared story? And so I’ve really been reflecting and looking deeper into what it means for people to commune in living rooms and spaces and tables and pubs and coffee shops.

 

And then boom, now I want to bring immersive technology. What does that look like? What does that mean? Does it come on the table? Is it lightweight? Is it screened? Is it shared? We desire the real human to human intimacy being in the same space, sharing the same air, looking at each other in the eye, and then allowing those things to then allow us to laugh together or to cry together and to be together. And so I, as a genre, I’ve been exploring those kind of key moments in looking at several new technologies that might open the door for that.

Charlie Melcher:

Well, Derek, it’s been such a pleasure to get to hang out with you again and talk, and thank you for sharing. And thank you for the beautiful work you do both as a creator and an educator and just can’t wait to see what new comes from the spaces and creativity of your brain. So thank you for being here.

Derek Ham:

Thank you. Thank you so much.

Charlie Melcher:

I am Charlie Melcher and I want to thank you for listening to the FoST podcast. The Future of StoryTelling is a passionate community of people who believe that better stories can create a better future. From our free monthly newsletter, FoST in Thought, to our annual FoST Explorers Club Membership, we’re constantly curating and celebrating the best of innovative story telling. To learn more and get involved, visit our website at fost.org. The FoST podcast is produced by Melcher Media in collaboration with our talented production partner Charts & Leisure. I hope to see you again soon for another deep dive into the world of story telling. Until then, please be safe, stay strong, and story on.